Buildroot 2011.11 released: details on new features

As planned, Buildroot 2011.11 has been released at the end of November. You can download this release as a tarball or through the Git repository.

This release brings a set of new features on which I thought it would be nice to give some details.

The file and local site method

Each package in Buildroot defines from where the source code for the particular component being built is fetched. Buildroot has of course always supported fetching a tarball from HTTP of FTP servers. Later on, Buildroot has added support for fetching from Git, Subversion and Bazaar repositories, for example by doing:

MYPKG_SITE = http://somelocation.com/svn/foobar/trunk
MYPKG_SITE_METHOD = svn

or

MYPKG_SITE = git://somelocation.com/foobar.git

The <pkg>_SITE_METHOD variable allows to define the fetching method. When not specified, Buildroot tries to guess it from the <pkg>_SITE value. Of course, in ambiguous cases such as Subversion or Git repositories over HTTP (as shown in the first example), the <pkg>_SITE_METHOD must be specified.

This new version of Buildroot brings two new site methods: file and local.

The file site method allows to specify the location of a local tarball as the source code for the component to be built. For example:

MYPKG_SITE = /opt/software/something-special-1.0.tar.gz
MYPKG_SITE_METHOD = file

This can be useful for internal software that isn’t publicly available on a HTTP or FTP server or in a revision control system. This new site method was added by David Wagner, who has been an intern at Bootlin between April and September this year.

The new local site method allows to specify the location of the source code to be built as a local directory. Buildroot will automatically copy the contents of this directory into the build directory of the component and build it from here. This is very useful because it allows to version control your source code as you wish, make changes to it, and easily tell Buildroot to rebuild your component. Note that the copy is using rsync so that further copies are very fast (see the pkg-reconfigure and pkg-rebuild targets below). An example of using the local site method:

MYPKG_SITE = /opt/software/something-special/
MYPKG_SITE_METHOD = local

This new site method has been implemented by myself, as the result from my experience of using Buildroot with various Bootlin customers.

The source directory override mechanism

The local site method described above is great for packaging special components that are specific to the embedded device one is working on, like the end-user application, or special internal libraries, etc.

However, there are cases where it is needed to work with a specific version of an open-source component. This is typically the case for the Linux kernel or the chosen bootloader (U-Boot, Barebox) or with other components. In that case, one may want to keep using Buildroot to build those components, but tell Buildroot to fetch the source code from a different location than the official tarball of the component. This is what the source directory override mechanism provide.

For example, if you want Buildroot to use the source code of the Linux kernel from /opt/project/linux/ rather than download it from a Git repository or as a tarball, you can write the following variable definition in a board/company/project/local.mk file:

LINUX_OVERRIDE_SRCDIR = /opt/project/linux

Then, you reference this file through the BR2_PACKAGE_OVERRIDE_FILE option, in Build options -> location of a package override file. When building the Linux kernel, Buildroot will copy the source code from /opt/project/linux into the kernel build directory, output/build/linux-VERSION/ and then start the build process of the kernel.

Basically, this mechanism is exactly like the local site method described previously, except that it is possible to override the source directory of a package without modifying the package .mk file, which is nice for open-source packages supported in Buildroot but that require local modifications.

To summarize, here is my recommendation on how to use Buildroot for packages that require project-specific modifications:

• You are using an existing open-source component on which you make some tiny bug fixes or modifications. In this case, the easiest solution is to add additional patches to the package directory, in package/<thepackage>/.
• You are using an existing open-source component, but are making major changes to it, that require proper version control outside of Buildroot. In this case, using the source directory override feature is recommended: it allows to keep the Buildroot package .mk file unmodified while still using your custom source code for the package.
• You have project-specific libraries or applications and want to integrate them in the build. My commendation is to version control them outside of Buildroot, and then create Buildroot packages for them using the local site method. Note that in the pkg_SITE variable, you can use the $(TOPDIR) variable to reference the top source directory of Buildroot. I for example often use MYAPP_SITE = $(TOPDIR)/../myapplication/.

The <pkg>-rebuild and <pkg>-reconfigure targets

For a long time, when one wanted to completely rebuild a given package from scratch, a possibility was has been to remove its build directory completely before restarting the build process:

rm -rf output/build/mypackage-1.0/
make

Or, using the -dirclean target available for each package:

make avahi-dirclean
make

As these commands remove completely the build directory, the build process is restarted from the beginning: extracting the source code, patching the source code, configuring, compiling, installing.

In 2011.11, we have added two new per-package targets to make it easy to use Buildroot during the development of components:

• make mypkg-reconfigure will restart the build process of mypkg from the configuration step (the source code is not re-extracted or repatched, so modifications made to the build directory are preserved)
• make mypkg-rebuild will restart the build process of mypkg from the compilation step (the source code is not re-extracted or repatched, the configuration step is not redone)

So, a typical usage could be:

emacs output/build/mypkg-1.0/src/foobar.c
make foobar-rebuild

However, beware that all build directories are removed when you do make clean, so the above example is only useful for quick testing of changes.

The case where the -reconfigure and -rebuild are really useful is in combination with the local site method or the source override directory mechanism. In this case, when pkg-reconfigure or pkg-rebuild is invoked, a synchronization of the source code is done between the source directory and the build directory is done before restarting the build.

Let’s take the example of a package named mypkg for which package/mypkg/mypkg.mk contains:

MYPKG_SITE = /opt/mypkg
MYPKG_SITE_METHOD = local

Then, to work on your package, you can simply do

emacs /opt/mypkg/foobar.c    # Edit as usual your project
make mypkg-rebuild           # Synchronizes the source code from
                             # /opt/mypkg to the build directory
                             # and restart the build

Integration of real-time extensions

In this 2011.11, an interesting addition is the integration of the Xenomai and RTAI real-time extensions to the Linux kernel. The Xenomai integration was initially proposed by Thomas de Schampheleire and then extended by myself, and I have also added the RTAI integration. This integration allows to seamlessly integrate the kernel patching process and the compilation of the required userspace libraries for those real-time extensions.

Conversion of the documentation to asciidoc

Back in 2004, one of my first contribution to Buildroot was to start writing documentation. At the time, the amount of documentation was small, so a single and simple HTML document was sufficient. Nowadays, Buildroot documentation has been extended significantly, and will have to be extended even further in the future. The approach of a single raw HTML document was no longer up to the task.

Therefore, I have worked on converting the existing documentation over to the asciidoc format. This allows us to split the source of the documentation in several files, for easier edition, and allows to generates a documentation in multiple formats: single HTML, split HTML, raw text or PDF.

Just run make manual in Buildroot 2011.11 to generate the manual. Note that the version available on the website is still the old HTML version, but it should soon be updated to the new asciidoc version.

Bootlin contributions

Bootlin has again contributed to this Buildroot release:

$ git shortlog -sen 2011.08..2011.11 | head -12
   126	Peter Korsgaard
   104	Gustavo Zacarias
    62	Thomas Petazzoni, from Bootlin
    27	Yann E. MORIN
    21	Sven Neumann
    13	Yegor Yefremov
    10	Thomas De Schampheleire
     7	H Hartley Sweeten
     5	Frederic Bassaler
     4	Arnout Vandecappelle (Essensium/Mind)
     4	Maxime Ripard, from Bootlin
     3	Baruch Siach

Our contributions have been:

• Implementation of the source directory override mechanism
• Implementation of the local and file site methods
• Implementation of the pkg-rebuild and pkg-reconfigure targets
• Conversion of the documentation to asciidoc and documentation improvements
• Various improvements for external toolchain support: optimization of the toolchain extraction and copy (reduced build time), integration of the support of the CodeSourcery x86 toolchains, update of all CodeSourcery toolchains to the latest available versions
• Removed useless arguments from the CMAKETARGETS, AUTOTARGETS and GENTARGETS macros, used by all packages in Buildroot. Instead, such pieces of information are automatically figured out from the package .mk file location in the source tree
• Added the cifs-utils package (for mounting CIFS network filesystems), the libplayer package, the picocom package.
• Cleanup, improve and merge the Xenomai integration done by Thomas de Schampheleire, and implement the RTAI integration
• Did a lot of cleanup in the source tree by creating a new support/ directory to contain various tools and scripts needed by Buildroot that were spread over the rest of the tree: the kconfig source code, the special libtool patches, various scripts, etc.

Next release cycle and next Buildroot meeting

The next release cycle has already started. After the meeting in Prague, it was decided that Peter Korsgaard (Buildroot maintainer) would maintain a next branch between the -rc1 and the final version of every release, in order to keep merging the new features for the next release while debugging the current release. This next branch for 2012.02 has already been merged. For example, the addition of the scp and Mercurial site methods has already been merged for 2012.02, as well as numerous other package updates.

On my side, besides usual package updates, I’d like to focus my work for this 2012.02 cycle on improving the testing coverage and on improving the documentation. My colleague Maxime Ripard is working on integrating systemd into Buildroot, as an alternate init mechanism.

The Buildroot community will also be organizing its next meeting in Brussels, on Friday February, 3rd 2012, right before the FOSDEM conference. Buildroot users and developers are invited to join, just contact us through the Buildroot mailing list.