Presentation of the Long Term Supported releases of Buildroot, a topic we also presented in a previous blog post
Appearance of the new top-level utils/ directory, containing various utilities directly useful for the Buildroot user, such as test-pkg, check-package, get-developers or scanpypi
Removal of $(HOST_DIR)/usr/, as everything has been moved up one level to $(HOST_DIR), to make the Buildroot SDK/toolchain more conventional
Document the new organization of the skeleton package, now split into several packages, to properly support various init systems. A new diagram has been added to clarify this topic.
List all package infrastructures that are available in Buildroot, since their number is growing!
Use SPDX license codes for licensing information in packages, which is now mandatory in Buildroot
Remove the indication that dependencies of host (i.e native) packages are derived from the dependencies of the corresponding package, since it’s no longer the case
Indicate that the check for hashes has been extended to also allow checking the hash of license files. This allows to detect changes in the license text.
Update the BR2_EXTERNAL presentation to cover the fact that multiples BR2_EXTERNAL trees are supported now.
Use the new relocatable SDK functionality that appeared in Buildroot 2017.08.
The practical labs have of course been updated to use Buildroot 2017.08, but also Linux 4.13 and U-Boot 2017.07, to remain current with upstream versions. In addition, they have been extended with two additional steps:
Booting the Buildroot generated system using TFTP and NFS, as an alternative to the SD card we normally use
Using genimage to generate a complete and ready to flash SD card image
Buildroot is a widely used embedded Linux build systems. A large number of companies and projects use Buildroot to produce customized embedded Linux systems for a wide range of embedded devices. Most of those devices are now connected to the Internet, and therefore subject to attacks if the software they run is not regularly updated to address security vulnerabilities.
The Buildroot project publishes a new release every three months, with each release providing a mix of new features, new packages, package updates, build infrastructure improvements… and security fixes. However, until earlier this year, as soon as a new version was published, the maintenance of the previous version stopped. This means that in order to stay up to date in terms of security fixes, users essentially had two options:
Update their Buildroot version regularly. The big drawback is that they get not only security updates, but also many other package updates, which may be problematic when a system is in production.
Stick with their original Buildroot version, and carefully monitor CVEs and security vulnerabilities in the packages they use, and update the corresponding packages, which obvisouly is a time-consuming process.
Starting with 2017.02, the Buildroot community has decided to offer one long term supported release every year: 2017.02 will be supported one year in terms of security updates and bug fixes, until 2018.02 is released. The usual three-month release cycle still applies, with 2017.05 and 2017.08 already being released, but users interested in a stable Buildroot version that is kept updated for security issues can stay on 2017.02.
Since 2017.02 was released on February 28th, 2017, six minor versions were published on a fairly regularly basis, almost every month, except in August:
With about 60 to 130 commits between each minor version, it is relatively easy for users to check what has been changed, and evaluate the impact of upgrading to the latest minor version to benefit from the security updates. The commits integrated in those minor versions are carefully chosen with the idea that users should be able to easily update existing systems.
In total, those six minor versions include 526 commits, of which 183 commits were security updates, representing roughly one third of the total number of commits. The other commits have been:
140 commits to fix build issues
57 commits to bump versions of packages for bug fixes. These almost exclusively include updates to the Linux kernel, using its LTS versions. For other packages, we are more conservative and generally don’t upgrade them.
17 commits to address issues in the licensing description of the packages
186 commits to fix miscellaneous issues, ranging from runtime issues affecting packages to bugs in the build infrastructure
The Buildroot community has already received a number of bug reports, patches or suggestions specifically targetting the 2017.02 LTS version, which indicates that developers and companies have started to adopt this LTS version.
Therefore, if you are interested in using Buildroot for a product, you should probably consider using the LTS version! We very much welcome feedback on this version, and help in monitoring the security vulnerabilities affecting software packages in Buildroot.
Over the last few releases, a significant number of improvements in terms of QA-related tooling has been done in the Buildroot project. As an embedded Linux build system, Buildroot has a growing number of packages, and maintaining all of those packages is a challenge. Therefore, improving the infrastructure around Buildroot to make sure that packages are in good shape is very important. Below we provide a summary of the different improvements that have been made.
Very much like the Linux kernel has a checkpatch.pl script to help contributors validate their patches, Buildroot now has a check-package script that allows to validate the coding style and check for common errors in Buildroot packages. Contributors are encouraged to use it to avoid common mistakes typically spotted during the review process.
check-package is capable of checking the .mk file, the Config.in, the .hash file of packages as well as the patches that apply to packages.
Contributed by Ricardo Martincoski, and written in Python, this tool will first appear in the next stable release 2017.05, to be published at the end of the month.
Buildroot’s page of package stats has been updated with a new column Warnings that lists the number of check-package issues to be fixed on each package. Not all packages have been fixed yet!
Besides coding style issues, another problem that the Buildroot community faces when accepting contributions of new packages or package updates, is that those contributions have rarely been tested on a large number of toolchain/architecture configurations. To help contributors in this testing, a test-pkg tool has been added.
Provided a Buildroot configuration snippet that enables the package to be tested, test-pkg tool will iterate over a number of toolchain/architecture configurations and make sure the package builds fine for all those configurations. The set of configurations being tested is the one used by Buildroot autobuilder’s infrastructure, which allows us to make sure that a package will not fail to build as soon as it is added into the tree. Contributors are therefore encouraged to use this tool for their package related contributions.
A very primitive version of this tool was originally contributed by Thomas Petazzoni, but it’s finally Yann E. Morin who took over, cleaned up the code, extended it and made it more generic, and contributed the final tool.
Runtime testing infrastructure
The Buildroot autobuilder infrastructure has been running for several years, and tests random configurations of Buildroot packages to make sure they build properly. This infrastructure allows the Buildroot developers to make sure that all combinations of packages build properly on all architectures, and has been a very useful tool to help increase Buildroot quality. However, this infrastructure does not perform any sort of runtime testing.
To address this, a new runtime testing infrastructure has recently been contributed to Buildroot by Thomas Petazzoni. Contrary to the autobuilder infrastructure that tests random configurations, this runtime testing infrastructure tests a well-defined set of configurations, and uses Qemu to make sure that they work properly. Located in support/testing this test infrastructure currently has only 25 test cases, but we plan to extend this over time with more and more tests.
For example, the ISO9660 test makes sure that Buildroot is capable of building an ISO9660 image that boots properly under Qemu. The test suite validates that it works in different configurations: using either Grub, Grub2 or Syslinux as a bootloader, and using a root filesystem entirely contained in an initramfs or inside the ISO9660 filesystem itself.
Besides doing run-time tests of packages, this infrastructure will also allow us to test various core Buildroot functionalities. We also plan to have the tests executed on a regular basis on a CI infrastructure such as Gitlab CI.
DEVELOPERS file and e-mail notification
Back in the 2016.11 release, we added a top-level file called DEVELOPERS, which plays more or less the same role as the Linux kernel’s MAINTAINERS file: associate parts of Buildroot, especially packages, with developers interested in this area. Since Buildroot doesn’t have a concept of per-package maintainers, we decided to simply call the file DEVELOPERS.
Thanks to the DEVELOPERS file, we are able to:
Provide the get-developers tool, which parses patches and returns a list of e-mail addresses to which the patches should be sent, very much like the Linux kernel get_maintainer.pl tool. This allows developers who have contributed a package to be notified when a patch is proposed for the same package, and get the chance to review/test the patch.
Notify the developers of a package of build failures caused by this package in the Buildroot autobuilder infrastructure. So far, all build results of this infrastructure were simply sent every day on the mailing list, which made it unpractical for individual developers to notice which build failures they should look at. Thanks to the DEVELOPERS file, we now send a daily e-mail individually to the developers whose packages are affected by build failures.
Notify the developers of the support for a given CPU architecture of build failures caused on the autobuilders for this architecture, in a manner similar to what is done for packages.
Thanks to this, we have seen developers who were not regularly following the Buildroot mailing list contribute again fixes for build failures caused by their packages, increasing the overall Buildroot quality. The DEVELOPERS file and the get-developers script, as well as the related improvements to the autobuilder infrastructure were contributed by Thomas Petazzoni.
Build testing of defconfigs on Gitlab CI
Buildroot contains a number of example configurations called defconfigs for various hardware platforms, which allow to build a minimal embedded Linux system, known to work on such platforms. At the time of this writing, Buildroot has 146 defconfigs for platforms ranging from popular development boards (Raspberry Pi, BeagleBone, etc.) to evaluation boards from SoC vendors and Qemu machine emulation.
In order to make sure all those defconfigs build properly, we used to have a job running on Travis CI, but we started to face limitations, especially in the maximum allowed build duration. Therefore, Arnout Vandecappelle migrated this on Gitlab CI and things have been running smoothly since then.
Before looking in more details at the contributions from Bootlin, let’s have a look at the main improvements provided by this release:
The big announcement is that 2017.02 is going to be a long term support release, maintained with security and other important fixes for one year. This will allow companies, users and projects that cannot upgrade at each Buildroot release to have a stable Buildroot version to work with, coming with regular updates for security and bug fixes. A few fixes have already been collected in the 2017.02.x branch, and regular point releases will be published.
Several improvements have been made to support reproducible builds, i.e the capability of having two builds of the same configuration provide the exact same bit-to-bit output. These are not enough to provide reproducible builds yet, but they are a piece of the puzzle, and more patches are pending for the next releases to move forward on this topic.
A package infrastructure for packages using the waf build system has been added. Seven packages in Buildroot are using this infrastructure currently.
Support for the OpenRISC architecture has been added, as well as improvements to the support of ARM64 (selection of ARM64 cores, possibility of building an ARM 32-bit system optimized for an ARM64 core).
The external toolchain infrastructure, which was all implemented in a single very complicated package, has been split into one package per supported toolchain and a common infrastructure. This makes it much easier to maintain.
A number of updates has been made to the toolchain components and capabilities: uClibc-ng bumped to 1.0.22 and enabled for ARM64, mips32r6 and mips64r6, gdb 7.12.1 added and switched to gdb 7.11 as the default, Linaro toolchains updated to 2016.11, ARC toolchain components updated to arc-2016.09, MIPS Codescape toolchains bumped to 2016.05-06, CodeSourcery AMD64 and NIOS2 toolchains bumped.
Eight new defconfigs for various hardware platforms have been added, including defconfigs for the NIOSII and OpenRISC Qemu emulation.
Sixty new packages have been added, and countless other packages have been updated or fixed.
More specifically, the contributions from Bootlin have been:
Thomas Petazzoni has handled the release of the first release candidate, 2017.02-rc1, and merged 742 patches out of the 1369 commits merged in this release.
Thomas contributed the initial work for the external toolchain infrastructure rework, which has been taken over by Romain Naour and finally merged thanks to Romain’s work.
Thomas contributed the rework of the ARM64 architecture description, to allow building an ARM 32-bit system optimized for a 64-bit core, and to allow selecting specific ARM64 cores.
Thomas contributed the raspberrypi-usbboot package, which packages a host tool that allows to boot a RaspberryPi system over USB.
Thomas fixed a large number of build issues found by the project autobuilders, contributing 41 patches to this effect.
Mylène Josserand contributed a patch to the X.org server package, fixing an issue with the i.MX6 OpenGL acceleration.
Gustavo Zacarias contributed a few fixes on various packages.
In addition, Bootlin sponsored the participation of Thomas to the Buildroot Developers meeting that took place after the FOSDEM conference in Brussels, early February. A report of this meeting is available on the eLinux Wiki.
The 2016.11 release of Buildroothas been published on November, 30th. The release announcement, by Buildroot maintainer Peter Korsgaard, gives numerous details about the new features and updates brought by this release. This new release provides support for using multiple BR2_EXTERNAL directories, gives some important updates to the toolchain support, adds default configurations for 9 new hardware platforms, and 38 new packages were added.
On a total of 1423 commits made for this release, Bootlin contributed a total of 253 commits:
Romain Perier contributed a package for the AMD Catalyst proprietary driver. Such drivers are usually not trivial to integrate, so having a ready-to-use package in Buildroot will really make it easier for Buildroot users who use hardware with an AMD/ATI graphics controller. This package provides both the X.org driver and the OpenGL implementation. This work was sponsored by one of Bootlin customer.
Gustavo Zacarias mainly contributed a large set of patches that do a small update to numerous packages, to make sure the proper environment variables are passed. This is a preparation change to bring top-level parallel build in Buildroot. This work was also sponsored by another Bootlin customer.
Thomas Petazzoni did contributions in various areas:
Added a DEVELOPERS file to the tree, to reference which developers are interested by which architectures and packages. Not only it allows the developers to be Cc’ed when patches are sent on the mailing list (like the get_maintainers script does), but it also used by Buildroot autobuilder infrastructure: if a package fails to build, the corresponding developer is notified by e-mail.
Misc updates to the toolchain support: switch to gcc 5.x by default, addition of gcc patches needed to fix various issues, etc.
Numerous fixes for build issues detected by Buildroot autobuilders
In addition to contributing 104 commits, Thomas Petazzoni also merged 1095 patches from other developers during this cycle, in order to help Buildroot maintainer Peter Korsgaard.
Finally, Bootlin also sponsored the Buildroot project, by funding the meeting location for the previous Buildroot Developers meeting, which took place in October in Berlin, after the Embedded Linux Conference. See the Buildroot sponsors page, and also the report from this meeting. The next Buildroot meeting will take place after the FOSDEM conference in Brussels.
In our effort to continuously improve and update our training materials, we have recently updated our Buildroot training course to Buildroot 2016.05, which was released at the end of May. In addition to adapting our practical labs to use this new Buildroot version, we have also improved the training materials to cover some of the new features that have been added over the last year in Buildroot. The most important changes are:
Cover the graph-size functionality, which allows to generate a pie chart of the filesystem size, per package. This is a very nice feature to analyze the size of your root filesystem and see how to reduce it.
Improve the description about the local site method and the override source directory functionalities, that are very useful when doing active application/library development in Buildroot, or to package custom application/library code.
Add explanations about using genimage to create complete SD card images that are ready to be flashed.
Add explanations about the hash file that can be added to packages to verify the integrity of the source code that is downloaded before it gets built.
For one of our customers building a product based on i.MX6 with a fairly low-volume, we had to design a mechanism to perform the factory flashing of each product. The goal is to be able to take a freshly produced device from the state of a brick to a state where it has a working embedded Linux system flashed on it. This specific product is using an eMMC as its main storage, and our solution only needs a USB connection with the platform, which makes it a lot simpler than solutions based on network (TFTP, NFS, etc.).
In order to achieve this goal, we have combined the imx-usb-loader tool with the fastboot support in U-Boot and some scripting. Thanks to this combination of a tool, running a single script is sufficient to perform the factory flashing, or even restore an already flashed device back to a known state.
The overall flow of our solution, executed by a shell script, is:
imx-usb-loader pushes over USB a U-Boot bootloader into the i.MX6 RAM, and runs it;
This U-Boot automatically enters fastboot mode;
Using the fastboot protocol and its support in U-Boot, we send and flash each part of the system: partition table, bootloader, bootloader environment and root filesystem (which contains the kernel image).
imx-usb-loader is a tool written by Boundary Devices that leverages the Serial Download Procotol (SDP) available in Freescale i.MX5/i.MX6 processors. Implemented in the ROM code of the Freescale SoCs, this protocol allows to send some code over USB or UART to a Freescale processor, even on a platform that has nothing flashed (no bootloader, no operating system). It is therefore a very handy tool to recover i.MX6 platforms, or as an initial step for factory flashing: you can send a U-Boot image over USB and have it run on your platform.
This tool already existed, we only created a package for it in the Buildroot build system, since Buildroot is used for this particular project.
Fastboot is a protocol originally created for Android, which is used primarily to modify the flash filesystem via a USB connection from a host computer. Most Android systems run a bootloader that implements the fastboot protocol, and therefore can be reflashed from a host computer running the corresponding fastboot tool. It sounded like a good candidate for the second step of our factory flashing process, to actually flash the different parts of our system.
Setting up fastboot on the device side
The well known U-Boot bootloader has limited support for this protocol:
The fastboot documentation in U-Boot can be found in the source code, in the doc/README.android-fastboot file. A description of the available fastboot options in U-Boot can be found in this documentation as well as examples. This gives us the device side of the protocol.
In order to make fastboot work in U-Boot, we modified the board configuration file to add the following configuration options:
Other options have to be selected, depending on the platform to fullfil the fastboot dependencies, such as USB Gadget support, GPT partition support, partitions UUID support or the USB download gadget. They aren’t explicitly defined anywhere, but have to be enabled for the build to succeed.
U-Boot enters the fastboot mode on demand: it has to be explicitly started from the U-Boot command line:
From now on, U-Boot waits over USB for the host computer to send fastboot commands.
Using fastboot on the host computer side
Fastboot needs a user-space program on the host computer side to talk to the board. This tool can be found in the Android SDK and is often available through packages in many Linux distributions. However, to make things easier and like we did for imx-usb-loader, we sent a patch to add the Android tools such as fastboot and adb to the Buildroot build system. As of this writing, our patch is still waiting to be applied by the Buildroot maintainers.
Thanks to this, we can use the fastboot tool to list the available fastboot devices connected:
# fastboot devices
Flashing eMMC partitions
For its flashing feature, fastboot identifies the different parts of the system by names. U-Boot maps those names to the name of GPT partitions, so your eMMC normally requires to be partitioned using a GPT partition table and not an old MBR partition table. For example, provided your eMMC has a GPT partition called rootfs, you can do:
# fastboot flash rootfs rootfs.ext4
To reflash the contents of the rootfs partition with the rootfs.ext4 image.
However, while using GPT partitioning is fine in most cases, i.MX6 has a constraint that the bootloader needs to be at a specific location on the eMMC that conflicts with the location of the GPT partition table.
To work around this problem, we patched U-Boot to allow the fastboot flash command to use an absolute offset in the eMMC instead of a partition name. Instead of displaying an error if a partition does not exists, fastboot tries to use the name as an absolute offset. This allowed us to use MBR partitions and to flash at defined offset our images, including U-Boot. For example, to flash U-Boot, we use:
The fastboot command must be explicitly called from the U-Boot prompt in order to enter fastboot mode. This is an issue for our use case, because the flashing process can’t be fully automated and required a human interaction. Using imx-usb-loader, we want to send a U-Boot image that automatically enters fastmode mode.
To achieve this, we modified the U-Boot configuration, to start the fastboot command at boot time:
Of course, this configuration is only used for the U-Boot sent using imx-usb-loader. The final U-Boot flashed on the device will not have the same configuration. To distinguish the two images, we named the U-Boot image dedicated to fastboot uboot_DO_NOT_TOUCH.
Putting it all together
We wrote a shell script to automatically launch the modified U-Boot image on the board, and then flash the different images on the eMMC (U-Boot and the root filesystem). We also added an option to flash an MBR partition table as well as flashing a zeroed file to wipe the U-Boot environment. In our project, Buildroot is being used, so our tool makes some assumptions about the location of the tools and image files.
Our script can be found here: flash.sh. To flash the entire system:
# ./flash.sh -a
To flash only certain parts, like the bootloader:
# ./flash.sh -b
By default, our script expects the Buildroot output directory to be in buildroot/output, but this can be overridden using the BUILDROOT environment variable.
By assembling existing tools and mechanisms, we have been able to quickly create a factory flashing process for i.MX6 platforms that is really simple and efficient. It is worth mentioning that we have re-used the same idea for the factory flashing process of the C.H.I.P computer. On the C.H.I.P, instead of using imx-usb-loader, we have used FEL based booting: the C.H.I.P indeed uses an Allwinner ARM processor, providing a different recovery mechanism than the one available on i.MX6.
Thomas Petazzoni, embedded Linux engineer and CTO at Bootlin, will be present during the FOSDEM conference. Thomas will also participate to the Buildroot Developers Meeting that will take place on February 1-2 in Brussels, hosted by Google.
The Buildroot project has recently released a new version, 2015.05. With exactly 1800 patches, it’s the largest release cycle ever, with patches from more than 100 different contributors. It’s an impressive number, showing the growing popularity of Buildroot as an embedded Linux build system.
The CHANGES file summarizes the most important improvements of this release.
Amongst those 1800 patches, 143 patches were contributed by Bootlin. Our most significant contributions for this release have been:
Addition of a package for the wf111 WiFi drivers. They allow to use a WiFi chip from Bluegiga, which is being used in one of our customer projects.
Addition of the support for using uClibc-ng. uClibc-ng is a “collaborative” fork of the uClibc project, which aims at doing more regular releases and have better testing. Maintained by Waldemar Brodkorb, the project has already seen several releases since its initial 1.0 release. Waldemar is merging patches from the original uClibc regularly, and adding more fixes. It allows Buildroot and other uClibc users to have well-identified uClibc stable versions instead of a 3 years old 0.9.33.2 version with dozens of patches on top of it. uClibc-ng is not currently used as the default uClibc version as of 2015.05, but it might very well be the case in 2015.08.
Important internal changes to the core infrastructure. Until this release, the make legal-info, make source, make external-deps and make source-check logic was relying only on the Buildroot configuration file. This was giving correct results for target packages which all have a corresponding Buildroot configuration option, but not for host packages (which for most of them don’t have Buildroot configuration options). Only a manual two-level dependency handling was done for the host packages for the above mentioned commands. With our work, the handling of those features has been moved to be part of the package infrastructure itself, so it’s using proper make recursivity to resolve the entire dependency tree. Due to this, the results of make legal-info or make external-deps may be longer following this release, but it’s because it’s now actually correct and complete. You can look at the patches for more details, but these changes are very deep into the core Buildroot infrastructure.
Addition of the imx-usb-loader package, which can be used to load over USB a new bootloader on i.MX6 platforms, even if the platform has no bootloader or a broken bootloader. We also use it as part of one of our customer projects.
With 142 patches, Bootlin engineer Thomas Petazzoni is the third contributor to this release by number of patches:
But by far, our most important contribution by far for this release is Thomas acting as the interim maintainer: on the total of 1800 patches merged for this release, Thomas has been the committer of 1446 patches. He has therefore been very active in merging the patches contributed by the Buildroot community.
Bootlin is a major contributor to the Buildroot upstream project, with more than 2800 patches merged as of May 2015 (including the ones contributed with our previous name). Our engineer Thomas Petazzoni alone has contributed more than 2700 patches. He has gathered an extensive knowledge of Buildroot and its internals, being one of the primary authors of the core infrastructures of Buildroot. He is a major participant to the Buildroot community, organizing the regular Buildroot Developer Days, supporting users through the mailing list and on IRC. Last but not least, Thomas acts as an interim maintainer when the main Buildroot maintainer is not available, an indication of Thomas strong involvement in the Buildroot project.
In addition, Bootlin has used and is using Buildroot in a significant number of customer projects, giving us an excellent view of Buildroot usage for real projects. This feedback has been driving some of our Buildroot contributions over the last years.
The 3 days training we have developed covers all the aspects of Buildroot: basic usage and configuration, understanding the source and build trees, creating new packages including advanced aspects, analyzing the build, tips for organizing your Buildroot work, using Buildroot for application development and more. See the detailed agenda.
We can deliver this training course anywhere in the world, at your location (see our rates and related details). We have also scheduled a first public session in English in Toulouse, France, on November 30 to December 2. Contact us at email@example.com if you are interested.
And finally, last but not least, like we do for all our training sessions, we are making the training materials freely available under a Creative Commons BY-SA license, at the time of the training announcement: the first session of this course is being given this week. For the Buildroot training, the available materials are:
Our materials have already been reviewed by some of the most prominent contributors to Buildroot: Peter Korsgaard (Buildroot maintainer), Yann E. Morin, Thomas De Schampheleire, Gustavo Zacarias and Arnout Vandecappelle. We would like to take this opportunity to thank them for their useful comments and suggestions in the development of this new training course.