Building a Linux system for the STM32MP1: basic system

As we announced recently, we are going to publish a series of blost post that describes how to build an embedded Linux device based on the STM32MP1 platform, using the Buildroot build system. In this first article, we are going to see how to create a basic Linux system, with minimal functionality. The hardware platform used in these articles is the STM32MP157-DK2.

What is Buildroot?

A Linux system is composed of a potentially large number of software components coming from different sources:

  • A bootloader, typically U-Boot, responsible for doing some minimal HW initialization, loading the Linux kernel and starting it
  • The Linux kernel itself, which implements features such as process management, memory management, scheduler, filesystems, networking stack and of course all device drivers for your hardware platform
  • User-space libraries and applications coming from the open-source community: command line tools, graphical libraries, networking libraries, cryptographic libraries, and more.
  • User-space libraries and applications developed internally, implementing the “business logic” of the embedded system

In order to assemble a Linux system with all those software components, one typically has two main choices:

  • Use a binary distribution, like Debian, Ubuntu or Fedora. Several of these distributions have support for the ARMv7 architecture. The main advantage of this solution is that it is easy: these binary distributions are familiar to most Linux users, they have a nice and easy-to-use package management system, all packages are pre-compiled so it is really fast to generate a Linux system. However, Linux systems generated this way are typically difficult to customize (software components are pre-built, so you cannot easily tweak their configuration to your needs) and difficult to optimize (in terms of footprint or boot time).
  • Use a build system, like Buildroot or Yocto/OpenEmbedded. These build systems build an entire Linux system from source code, which means that it can be highly customized and optimized to your needs. Of course, it is less simple than using a binary distribution and because you are building all components from source code, a non-negligible amount of CPU time will be spent on compiling code.

BuildrootIn this series of blog post, we have chosen to use Buildroot, which is an easy-to-use build system, which is a good match for engineers getting started with embedded Linux. For more general details about Buildroot, you can read the freely available training materials of our Embedded Linux development with Buildroot training course.

Buildroot is a set of Makefiles and script that automates the process of download the source code of the different software components, extract them, configure them, build them and install them. It ultimately generates a system image that is ready to be flashed, and which typically contains the bootloader, the Linux kernel image and the root filesystem. It is important to understand that Buildroot itself does not contain the source code for Linux, U-Boot or any other component: it is only a set of scripts/recipes that describes where to download the source code from, and how to build it.

Principle of an embedded Linux build system

Building the minimal system with Buildroot

Let’s started by getting the source of Buildroot from its upstream Git repository:

git clone git://git.buildroot.net/buildroot
cd buildroot

Starting a Buildroot configuration is then typically done by running make menuconfig, and then selecting all the relevant options for your system. Here, we are instead going to use a pre-defined configuration that we created for the STM32MP157-DK2 platform. This pre-defined configuration has been submitted to the upstream Buildroot project, but has not yet been merged as of this writing, so we’ll use an alternate Git branch:

git remote add tpetazzoni https://github.com/tpetazzoni/buildroot.git
git fetch tpetazzoni
git checkout -b stm32mp157-dk2 tpetazzoni/2019.02/stm32mp157-dk

The 2019.02/stm32mp157-dk branch in your author’s Buildroot Git repository is based on upstream Buildroot 2019.02.x branch and contains 4 additional patches needed to support the STM32MP157-DK2 platform.

Let’s continue by telling Buildroot to load the pre-defined configuration for the STM32MP157-DK2:

make stm32mp157_dk_defconfig

We could start the build right away, as this configuration works fine, but to illustrate how to modify the configuration (and speed up the build!) we will adjust one aspect of the system configuration. To do so, let’s run Buildroot’s menuconfig. People who have already configured the Linux kernel should be familiar with the tool, as it is the exact same configuration utility.

make menuconfig

At this point, if the command fails due to the ncurses library being missing, make sure to install the libcnurses-dev or ncurses-devel package on your Linux distribution (the exact package name depends on the distribution you’re using).

Once in menuconfig, go to the Toolchain sub-menu. By default the Toolchain type is Buildroot toolchain. Change it to External toolchain by pressing the Enter key. When Buildroot toolchain is selected, Buildroot builds its own cross-compiler, which takes quite some time. Selecting External toolchain tells Buildroot to use a pre-existing cross-compiler, which in our case is the one provided by ARM for the ARMv7 architecture.

Exit menuconfig and save the configuration. It is now time to start the build by running make. However, your author generally likes to keep the output of the build in a log file, using the following incantation:

make 2>&1 | tee build.log

Now that Buildroot starts by checking if your system has a number of required packages installed, and will abort if not. Please follow section System requirements > Mandatory packages of the Buildroot manual to install all the appropriate dependencies. Restart the make command once all dependencies have been installed.

The build process took 10 minutes on your author’s machine. All the build output is conveniently grouped in the sub-directory named output/, in which the most important results are in output/images/:

  • output/images/zImage is the Linux kernel image
  • output/images/stm32mp157c-dk2.dtb is the Device Tree Blob, i.e the piece of data that describes to the Linux kernel the hardware it is running on. We’ll talk more about Device Tree in the second blog post of this series
  • output/images/rootfs.{ext4,ext2} is the image of the root filesystem, i.e the filesystem that contains all the user-space libraries and applications. It’s using the ext4 filesystem format, which is the de-facto standard filesystem format in Linux for block storage.
  • output/images/u-boot-spl.stm32 is the first stage bootloader
  • output/images/u-boot.img is the second stage bootloader
  • output/images/sdcard.img is a complete, ready-to-use SD card image, which was generated from the previous images

Flashing and testing the system

First things first, we’ll need to write sdcard.img to a microSD card:

sudo dd if=output/images/sdcard.img of=/dev/mmcblk0 bs=1M conv=fdatasync status=progress

Of course, make sure that, on your system, the microSD card is really identified as /dev/mmcblk0. And beware that all the data on your microSD card will be lost!

Insert the microSD card in the microSD card connector of the STM32MP157-DK2 board, i.e connector CN15.

Connect a USB to micro-USB cable between your PC and the connector labeled ST-LINK CN11 on the board. A device called /dev/ttyACM0 will appear on your PC, through which you’ll be able to access the board’s serial port. Install and run a serial port communication program on your PC, your author’s favorite is the very minimalistic picocom:

picocom -b 115200 /dev/ttyACM0

Finally, power up the board by connecting a USB-C cable to connector PWR_IN CN6. You should then see a number of messages on the serial port, all the way up to Buildroot login:. You can then login with the root user, no password will be requested.

STM32MP157-DK2 in situation

How is the system booting ?

Let’s look at the main steps of the boot process, by studying the boot log visible on the serial port:

U-Boot SPL 2018.11-stm32mp-r2.1 (Apr 24 2019 - 10:37:17 +0200)

This message is printed by the first stage bootloader, i.e the code contained in the file u-boot-spl.stm32, compiled as part of the U-Boot bootloader. This first stage bootloader is directly loaded by the STM32MP157 system-on-chip. This first stage bootloader must be small enough to fit inside the STM32MP157 internal memory.

U-Boot 2018.11-stm32mp-r2.1 (Apr 24 2019 - 10:37:17 +0200)

This message is printed by the second stage bootloader, which was loaded from storage into external memory by the first stage bootloader. This second stage bootloader is the file u-boot.img, which was also compiled as part of the U-Boot bootloader.

Retrieving file: /boot/zImage
Retrieving file: /boot/stm32mp157c-dk2.dtb

These messages are printed by the second stage bootloader: we see it is loading the Linux kernel image (file zImage) and the Device Tree Blob describing our hardware platform (file stm32mp157c-dk2.dtb). It indicates that U-Boot has loaded both files into memory: it is now ready to start the Linux kernel.

Starting kernel ...

This is the last message printed by U-Boot before jumping into the kernel.

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.19.26 (thomas@windsurf) (gcc version 8.2.1 20180802 (GNU Toolchain for the A-profile Architecture 8.2-2018.11 (arm-rel-8.26))) #1 SMP PREEMPT Wed Apr 24 10:38:00 CEST 2019

And immediately after that, we have the first messages of the Linux kernel, showing the version of Linux and the date/time it was built. Numerous other kernel messages are then displayed, until:

[    3.248315] VFS: Mounted root (ext4 filesystem) readonly on device 179:4.

This message indicates that the kernel has mounted the root filesystem. After this point, the kernel will start the first user-space process, so the next messages are user-space services being initialized:

Starting syslogd: OK
[...]
Welcome to Buildroot
buildroot login: 

Until we reach a login prompt.

Exploring the system

After logging in as root, you have access to a regular Linux shell, with most basic Linux commands available. You can run ps to see the processes, run ls / to see the contents of the root filesystem, etc.

You can also play a bit with the hardware, for example to turn on and off one of the LEDs of the board:

echo 255 > /sys/class/leds/heartbeat/brightness
echo 0 > /sys/class/leds/heartbeat/brightness

Understanding the Buildroot configuration

So far, we have used a pre-defined Buildroot configuration, without really understanding what it does and how it built this basic system for our board. So let’s go back in make menuconfig and see how Buildroot was configured.

In the Target options menu, obviously the ARM Little Endian architecture was chosen, and more specifically Cortex-A7 was chosen as the Target Architecture Variant. Indeed the entire Linux system runs on the Cortex-A7 cores.

In the Build options menu, nothing was changed from the default values.

In the Toolchain menu, we previously modified to use an External toolchain to use a pre-existing cross-compiler and save on build time. All other options were kept as their default.

In the System configuration menu, we defined the following things:

  • Root filesystem overlay directories is set to board/stmicroelectronics/stm32mp157-dk/overlay/. This option tells Buildroot that the contents of the board/stmicroelectronics/stm32mp157-dk/overlay/ directory must be copied into the root filesystem at the end of the build. It allows to add custom files to the root filesystem.
  • Custom scripts to run after creating filesystem images is set to support/scripts/genimage.sh and the related option Extra arguments passed to custom scripts is set to -c board/stmicroelectronics/stm32mp157-dk/genimage.cfg. This tells Buildroot to call this genimage.sh script at the very end of the build: its purpose is to generate the final SD card image we have used.

In the Kernel menu, we have obviously configured which Linux kernel version and configuration should be used:

  • We are downloading the Linux kernel source code as a tarball from Github, using a custom Buildroot macro called github. Based on this information, Buildroot will go to the Git repository at https://github.com/STMicroelectronics/linux/, and get the kernel version identified by the tag v4.19-stm32mp-r1.2
  • Configuration file path is set to board/stmicroelectronics/stm32mp157-dk/linux.config. This is the file that contains the kernel configuration. We have prepared a custom kernel configuration to have a simple but working kernel configuration. Of course, it can be adjusted to your needs, as we will demonstrate in the next blog post.
  • We enabled the option Build a Device Tree Blob (DTB) and set In-tree Device Tree Source file names to stm32mp157c-dk2. This tells Buildroot to build and install the Device Tree Blob that matches our hardware platform.
  • Finally, we enabled Install kernel image to /boot in target, so that the kernel image and the Device Tree blob are installed inside the /boot directory in the root filesystem. Indeed, our U-Boot configuration will load them from here (see below).

In the Target packages menu, we have kept the default: only the BusyBox package is enabled. BusyBox is a very popular tool in the embedded Linux ecosystem: it provides a lightweight replacement for a Linux shell and most common Linux command line tools (cp, mv, ls, vi, wget, tar, and more). Our basic system in fact only contains BusyBox!

In the Filesystem images menu, we have enabled the ext2/3/4 root filesystem type and chosen the ext4 variant. As explained above, ext4 is kind of the de-facto standard Linux filesystem for block storage devices such as SD cards.

In the Bootloaders menu, we enabled U-Boot, where a significant number of options need to be tweaked:

  • We download U-Boot from a STMicroelectronics Git repository at https://github.com/STMicroelectronics/u-boot.git and use the Git tag v2018.11-stm32mp-r2.1.
  • This U-Boot comes with a pre-defined configuration called stm32mp15_basic, which we select using Board defconfig.
  • However, it turns out that this pre-defined configuration enables the STM32 watchdog, and since our Linux user-space does not have a watchdog daemon to tick the watchdog regularly, it would reset constantly. Using a small additional snippet of U-Boot configuration, stored in the file board/stmicroelectronics/stm32mp157-dk/uboot-fragment.config, we disable the watchdog. Of course, it should be re-enabled and properly handled in Linux user-space for a final product.
  • In the U-Boot binary format sub-menu, we tell Buildroot that the second stage bootloader image will be called u-boot.img, and this is the one Buildroot should install in output/images
  • We tell Buildroot that our U-Boot configuration will build a first stage bootloader called spl/u-boot-spl.stm32, which allows Buildroot to install it in output/images
  • Finally, we pass a custom DEVICE_TREE=stm32mp157c-dk2 option in the U-Boot environment, which is needed for the U-Boot build process to find the Device Tree used internally by U-Boot.

Finally, in the Host utilities menu, we enable the host genimage package.

This entire configuration is saved in a simple text file called configs/stm32mp157_dk_defconfig, which is the one we loaded initially when running make stm32mp157_dk_defconfig. We suggest you take a moment to look at configs/stm32mp157_dk_defconfig and see the configuration options it defines.

What happens during the Buildroot build?

With all these options in place, here is what Buildroot has done to build our system (we have omitted some intermediate steps or package dependencies for the sake of brievity):

  1. Download and install the pre-built ARM compiler from ARM’s website, and install the C and C++ libraries inside the target root filesystem
  2. Download the Linux kernel source code from STMicroelectronics Github repository, configure it with our configuration file, build it, install zImage and stm32mp157c-dk2.dtb both in output/images and in the target root filesystem in the /boot directory. It also installs the Linux kernel modules inside the target root filesystem
  3. Download the U-Boot source code from STMicroelectronics Github repository, configure it, build it and install u-boot-spl.stm32 and u-boot.img in output/images
  4. Download the Busybox source code from the project official website, configure it, build it and install it inside the target root filesystem.
  5. Copies the contents of the rootfs overlay inside the target root filesystem
  6. Produce the ext4 image of the root filesystem, and install it as output/images/rootfs.ext4
  7. Call the genimage.sh script, whose purpose is to generate the final SD card image, output/images/sdcard.img

Let’s now have a look at the file board/stmicroelectronics/stm32mp157-dk/genimage.cfg, which tells the genimage utility how to create the final SD card image:

image sdcard.img {
        hdimage {
                gpt = "true"
        }

        partition fsbl1 {
                image = "u-boot-spl.stm32"
        }

        partition fsbl2 {
                image = "u-boot-spl.stm32"
        }

        partition uboot {
                image = "u-boot.img"
        }

        partition rootfs {
                image = "rootfs.ext4"
                partition-type = 0x83
                bootable = "yes"
                size = 256M
        }
}

What this file says is:

  • We want to create a file named sdcard.img
  • This file will contain a number of partitions, described by a GPT partition table. This is necessary for the STM32MP157 built-in ROM code to find the first stage bootloader.
  • The first two partitions are named fsbl1 and fsbl2, and contain the raw binary of the first stage bootloader, i.e there is no filesystem in those partitions. it is the STM32MP157 built-in ROM code that is hardcoded to search the first stage bootloader in the first two partitions whose name start with fsbl.
  • The third partition named uboot contains the second stage bootloader, also as a raw binary (no filesystem). Indeed, the first stage bootloader is configured to search the second bootloader from the third partition of the SD card (this is defined in the U-Boot configuration and can be modified if needed)
  • The fourth partition contains the ext4 filesystem image that Buildroot has produced, which is in fact our Linux root filesystem, with BusyBox, the standard C/C++ libraries and the Linux kernel image and Device Tree Blob.

This last partition is marked bootable. This is important because the U-Boot configuration for the STM32MP157 hardware platform by default uses the U-Boot Generic Distro Concept. At boot, U-Boot will search for the partition marked bootable, and then inside the filesystem contained in this partition, look for the file /boot/extlinux/extlinux.conf to know how to boot the system.

This file extlinux.conf is inside our root filesystem overlay at board/stmicroelectronics/stm32mp157-dk/overlay/boot/extlinux/extlinux.conf, so it is installed in our root filesystem as /boot/extlinux/extlinux.conf so that U-Boot finds it. This file simply contains:

label stm32mp15-buildroot
  kernel /boot/zImage
  devicetree /boot/stm32mp157c-dk2.dtb
  append root=/dev/mmcblk0p4 rootwait

Which tells U-Boot that the kernel image to load is /boot/zImage, that the Device Tree Blob to use is /boot/stm32mp157c-dk2.dtb and that the string root=/dev/mmcblk0p4 rootwait must be passed as arguments to the Linux kernel when booting. The root=/dev/mmcblk0p4 is particularly important, because it is the one telling the Linux kernel where the root filesystem is located.

So, if we summarize the boot process of our hardware platform with those new details in mind, it looks like this:

  1. The STM32MP157 built-in ROM code looks for the GPT partitions whose name start with fsbl, and if one is found, loads the contents into the STM32 internal memory and runs it. This is our first stage bootloader.
  2. This first stage bootloader is hard-coded to load the second stage bootloader from the third partition of the SD card. So it initializes the external RAM, loads this second stage bootloader into external RAM and runs it.
  3. The second stage bootloader does some more initialization, and then looks for a partition marked bootable. It finds that the fourth partition is bootable. It loads the /boot/extlinux/extlinux.conf file, thanks to which it learns where the kernel and Device Tree are located. It loads both, and starts the kernel with the arguments also specified in the extlinux.conf file.
  4. The Linux kernel runs, up to the point where it mounts the root filesystem, whose location is indicated by the root=/dev/mmcblk0p4 argument. After mounting the root filesystem, the kernel starts the first user-space process.
  5. The first user-space process that runs is /sbin/init, implemented by BusyBox. It starts a small number of services, and then starts the login prompt.

Conclusion

In this long initial blog post, we have learned what Buildroot is, how to use it to build a basic system for the STM32MP157 platform, how the Buildroot configuration was created, and how the STM32MP157 platform is booting.

Stay tuned for the next blog post, during which we will learn how to plug an additional device to the board: a pressure, temperature and humdity sensor connected over I2C, and how to make it work with Linux.

Bootlin at the Buildroot Developers Meeting, February 2019

Buildroot logoIt’s now a tradition: the Buildroot project organizes one of its Buildroot Developers Meeting right after the FOSDEM conference. 2019 was no exception, and the meeting took place from February 4 to February 6, a three days duration, instead of the traditional two days duration from the previous years. Once again, the meeting was sponsored by Google, who provided the meeting location and lunch for all participants. Bootlin participated to the event, by allowing its engineer Thomas Petazzoni to join the meeting.

The meeting was a mix of discussions on various topics and actual hacking, with a focus on reducing the backlog of pending patches. The report synthetizes the most important discussion items:

  • A short general assembly of the Buildroot Association took place
  • Some discussions around the download infrastructure took place, related to the re-introduction of the make source-check feature and the issue of tarball reproducibility with version control system download backends
  • Discussion about introducing Config.in options for all host packages, an idea that we decided to not pursue for the moment.
  • Discussion about the instrumentation hooks that are used to collect the list of files installed by packages, and how we can achieve this goal in a way that is both efficient and reliable
  • Discussion on which Qt5 versions to support
  • Discussion on participating to the Google Summer of Code. We wrote a few topic ideas and applied as an organization for GSoC 2019.
  • Discussion on how to integrate support for systemd sysusers mechanism

Reading the work on the pending patches, we managed to reduce the backlog from about 300 patches to around 170 patches, which is a very significant achievement.

From left to right: Mark Corbin, Adam Duskett, Angelo Compagnucci (front), Peter Korsgaard (back), Thomas Petazzoni (front), Arnout Vandecappelle (back), Thomas De Schampheleire, Adrian Perez de Castro and Titouan Christophe. Behind the camera: Yann E. Morin.

More specifically, Thomas Petazzoni took advantage of this meeting to:

  • Finalize his work on the pkg-stats script, to include details about the latest available upstream version of each Buildroot package. To do so, it relies on information provided by the release-monitoring.org website. The information is not yet accurate for all packages, but the accuracy can be improved by contributing to release-monitoring.org. The updated package statistics page now provides those details, which will help ensure Buildroot packages are kept up-to-date.
  • Review in detail the patch series from Adam Duskett introducing support for GObject Introspection. It is far from a trivial package due to the need to run during the build some small binaries using Qemu. While the series is not merged yet, we have a much better understanding of it, which will help complete the review process.
  • Do a final review and apply the lengthy patch series reworking the fftw package.
  • Participate, as a Buildroot co-maintainer, to the pending patches backlog cleanup, by reviewing and/or merging a significant number of patches.

It was once again a very nice and productive meeting. The next meeting will take place as usual around the Embedded Linux Conference Europe, in October, in Lyon (France).

Back from ELCE 2017: slides and videos

Bootlin participated to the Embedded Linux Conference Europe last week in Prague. With 7 engineers attending, 4 talks, one BoF and a poster at the technical showcase, we had a strong presence to this major conference of the embedded Linux ecosystem. All of us had a great time at this event, attending interesting talks and meeting numerous open-source developers.

Bootlin team at the Embedded Linux Conference Europe 2017
Bootlin team at the Embedded Linux Conference Europe 2017. Top, from left to right: Maxime Ripard, Grégory Clement, Boris Brezillon, Quentin Schulz. Bottom, from left to right: Miquèl Raynal, Thomas Petazzoni, Michael Opdenacker.

In this first blog post about ELCE, we want to share the slides and videos of the talks we have given during the conference.

SD/eMMC: New Speed Modes and Their Support in Linux – Gregory Clement

Grégory ClementSince the introduction of the original “default”(DS) and “high speed”(HS) modes, the SD card standard has evolved by introducing new speed modes, such as SDR12, SDR25, SDR50, SDR104, etc. The same happened to the eMMC standard, with the introduction of new high speed modes named DDR52, HS200, HS400, etc. The Linux kernel has obviously evolved to support these new speed modes, both in the MMC core and through the addition of new drivers.

This talk will start by introducing the SD and eMMC standards and how they work at the hardware level, with a specific focus on the new speed modes. With this hardware background in place, we will then detail how these standards are supported by Linux, see what is still missing, and what we can expect to see in the future.

Slides [PDF], Slides [LaTeX source]

An Overview of the Linux Kernel Crypto Subsystem – Boris Brezillon

Boris BrezillonThe Linux kernel has long provided cryptographic support for in-kernel users (like the network or storage stacks) and has been pushed to open these cryptographic capabilities to user-space along the way.

But what is exactly inside this subsystem, and how can it be used by kernel users? What is the official userspace interface exposing these features and what are non-upstream alternatives? When should we use a HW engine compared to a purely software based implementation? What’s inside a crypto engine driver and what precautions should be taken when developing one?

These are some of the questions we’ll answer throughout this talk, after having given a short introduction to cryptographic algorithms.

Slides [PDF], Slides [LaTeX source]

Buildroot: What’s New? – Thomas Petazzoni

Thomas PetazzoniBuildroot is a popular and easy to use embedded Linux build system. Within minutes, it is capable of generating lightweight and customized Linux systems, including the cross-compilation toolchain, kernel and bootloader images, as well as a wide variety of userspace libraries and programs.

Since our last “What’s new” talk at ELC 2014, three and half years have passed, and Buildroot has continued to evolve significantly.

After a short introduction about Buildroot, this talk will go through the numerous new features and improvements that have appeared over the last years, and show how they can be useful for developers, users and contributors.

Slides [PDF], Slides [LaTeX source]

Porting U-Boot and Linux on New ARM Boards: A Step-by-Step Guide – Quentin Schulz

May it be because of a lack of documentation or because we don’t know where to look or where to start, it is not always easy to get started with U-Boot or Linux, and know how to port them to a new ARM platform.

Based on experience porting modern versions of U-Boot and Linux on a custom Freescale/NXP i.MX6 platform, this talk will offer a step-by-step guide through the porting process. From board files to Device Trees, through Kconfig, device model, defconfigs, and tips and tricks, join this talk to discover how to get U-Boot and Linux up and running on your brand new ARM platform!

Slides [PDF], Slides [LaTeX source]

BoF: Embedded Linux Size – Michael Opdenacker

This “Birds of a Feather” session will start by a quick update on available resources and recent efforts to reduce the size of the Linux kernel and the filesystem it uses.

An ARM based system running the mainline kernel with about 3 MB of RAM will also be demonstrated.

If you are interested in the size topic, please join this BoF and share your experience, the resources you have found and your ideas for further size reduction techniques!

Slides [PDF], Slides [LaTeX source]

Buildroot training course updated to Buildroot 2017.08

BuildrootBack in June 2015, we announced the availability of a training course on Buildroot, a popular and easy to use embedded Linux build system. A year later, we updated it to cover Buildroot 2016.05. We are happy to announce a new update: we now cover Buildroot 2017.08.

The most significant updates are:

  • Presentation of the Long Term Supported releases of Buildroot, a topic we also presented in a previous blog post
  • Appearance of the new top-level utils/ directory, containing various utilities directly useful for the Buildroot user, such as test-pkg, check-package, get-developers or scanpypi
  • Removal of $(HOST_DIR)/usr/, as everything has been moved up one level to $(HOST_DIR), to make the Buildroot SDK/toolchain more conventional
  • Document the new organization of the skeleton package, now split into several packages, to properly support various init systems. A new diagram has been added to clarify this topic.
  • List all package infrastructures that are available in Buildroot, since their number is growing!
  • Use SPDX license codes for licensing information in packages, which is now mandatory in Buildroot
  • Remove the indication that dependencies of host (i.e native) packages are derived from the dependencies of the corresponding package, since it’s no longer the case
  • Indicate that the check for hashes has been extended to also allow checking the hash of license files. This allows to detect changes in the license text.
  • Update the BR2_EXTERNAL presentation to cover the fact that multiples BR2_EXTERNAL trees are supported now.
  • Use the new relocatable SDK functionality that appeared in Buildroot 2017.08.

The practical labs have of course been updated to use Buildroot 2017.08, but also Linux 4.13 and U-Boot 2017.07, to remain current with upstream versions. In addition, they have been extended with two additional steps:

  • Booting the Buildroot generated system using TFTP and NFS, as an alternative to the SD card we normally use
  • Using genimage to generate a complete and ready to flash SD card image

We will be delivering this course to one of our customers in Germany next month, and are of course available to deliver it on-site anywhere in the world if you’re interested! And of course, we continue to publish, for free, all the materials used in this training session: slides and labs.

Buildroot Long Term Support releases: from 2017.02 to 2017.02.6 and counting

Buildroot LogoBuildroot is a widely used embedded Linux build systems. A large number of companies and projects use Buildroot to produce customized embedded Linux systems for a wide range of embedded devices. Most of those devices are now connected to the Internet, and therefore subject to attacks if the software they run is not regularly updated to address security vulnerabilities.

The Buildroot project publishes a new release every three months, with each release providing a mix of new features, new packages, package updates, build infrastructure improvements… and security fixes. However, until earlier this year, as soon as a new version was published, the maintenance of the previous version stopped. This means that in order to stay up to date in terms of security fixes, users essentially had two options:

  1. Update their Buildroot version regularly. The big drawback is that they get not only security updates, but also many other package updates, which may be problematic when a system is in production.
  2. Stick with their original Buildroot version, and carefully monitor CVEs and security vulnerabilities in the packages they use, and update the corresponding packages, which obvisouly is a time-consuming process.

Starting with 2017.02, the Buildroot community has decided to offer one long term supported release every year: 2017.02 will be supported one year in terms of security updates and bug fixes, until 2018.02 is released. The usual three-month release cycle still applies, with 2017.05 and 2017.08 already being released, but users interested in a stable Buildroot version that is kept updated for security issues can stay on 2017.02.

Since 2017.02 was released on February 28th, 2017, six minor versions were published on a fairly regularly basis, almost every month, except in August:

With about 60 to 130 commits between each minor version, it is relatively easy for users to check what has been changed, and evaluate the impact of upgrading to the latest minor version to benefit from the security updates. The commits integrated in those minor versions are carefully chosen with the idea that users should be able to easily update existing systems.

In total, those six minor versions include 526 commits, of which 183 commits were security updates, representing roughly one third of the total number of commits. The other commits have been:

  • 140 commits to fix build issues
  • 57 commits to bump versions of packages for bug fixes. These almost exclusively include updates to the Linux kernel, using its LTS versions. For other packages, we are more conservative and generally don’t upgrade them.
  • 17 commits to address issues in the licensing description of the packages
  • 186 commits to fix miscellaneous issues, ranging from runtime issues affecting packages to bugs in the build infrastructure

The Buildroot community has already received a number of bug reports, patches or suggestions specifically targetting the 2017.02 LTS version, which indicates that developers and companies have started to adopt this LTS version.

Therefore, if you are interested in using Buildroot for a product, you should probably consider using the LTS version! We very much welcome feedback on this version, and help in monitoring the security vulnerabilities affecting software packages in Buildroot.

Recent improvements in Buildroot QA

Over the last few releases, a significant number of improvements in terms of QA-related tooling has been done in the Buildroot project. As an embedded Linux build system, Buildroot has a growing number of packages, and maintaining all of those packages is a challenge. Therefore, improving the infrastructure around Buildroot to make sure that packages are in good shape is very important. Below we provide a summary of the different improvements that have been made.

Buildroot

check-package script

Very much like the Linux kernel has a checkpatch.pl script to help contributors validate their patches, Buildroot now has a check-package script that allows to validate the coding style and check for common errors in Buildroot packages. Contributors are encouraged to use it to avoid common mistakes typically spotted during the review process.

check-package is capable of checking the .mk file, the Config.in, the .hash file of packages as well as the patches that apply to packages.

Contributed by Ricardo Martincoski, and written in Python, this tool will first appear in the next stable release 2017.05, to be published at the end of the month.

Buildroot’s page of package stats has been updated with a new column Warnings that lists the number of check-package issues to be fixed on each package. Not all packages have been fixed yet!

test-pkg script

Besides coding style issues, another problem that the Buildroot community faces when accepting contributions of new packages or package updates, is that those contributions have rarely been tested on a large number of toolchain/architecture configurations. To help contributors in this testing, a test-pkg tool has been added.

Provided a Buildroot configuration snippet that enables the package to be tested, test-pkg tool will iterate over a number of toolchain/architecture configurations and make sure the package builds fine for all those configurations. The set of configurations being tested is the one used by Buildroot autobuilder’s infrastructure, which allows us to make sure that a package will not fail to build as soon as it is added into the tree. Contributors are therefore encouraged to use this tool for their package related contributions.

A very primitive version of this tool was originally contributed by Thomas Petazzoni, but it’s finally Yann E. Morin who took over, cleaned up the code, extended it and made it more generic, and contributed the final tool.

Runtime testing infrastructure

The Buildroot autobuilder infrastructure has been running for several years, and tests random configurations of Buildroot packages to make sure they build properly. This infrastructure allows the Buildroot developers to make sure that all combinations of packages build properly on all architectures, and has been a very useful tool to help increase Buildroot quality. However, this infrastructure does not perform any sort of runtime testing.

To address this, a new runtime testing infrastructure has recently been contributed to Buildroot by Thomas Petazzoni. Contrary to the autobuilder infrastructure that tests random configurations, this runtime testing infrastructure tests a well-defined set of configurations, and uses Qemu to make sure that they work properly. Located in support/testing this test infrastructure currently has only 25 test cases, but we plan to extend this over time with more and more tests.

For example, the ISO9660 test makes sure that Buildroot is capable of building an ISO9660 image that boots properly under Qemu. The test suite validates that it works in different configurations: using either Grub, Grub2 or Syslinux as a bootloader, and using a root filesystem entirely contained in an initramfs or inside the ISO9660 filesystem itself.

Besides doing run-time tests of packages, this infrastructure will also allow us to test various core Buildroot functionalities. We also plan to have the tests executed on a regular basis on a CI infrastructure such as Gitlab CI.

DEVELOPERS file and e-mail notification

Back in the 2016.11 release, we added a top-level file called DEVELOPERS, which plays more or less the same role as the Linux kernel’s MAINTAINERS file: associate parts of Buildroot, especially packages, with developers interested in this area. Since Buildroot doesn’t have a concept of per-package maintainers, we decided to simply call the file DEVELOPERS.

Thanks to the DEVELOPERS file, we are able to:

  • Provide the get-developers tool, which parses patches and returns a list of e-mail addresses to which the patches should be sent, very much like the Linux kernel get_maintainer.pl tool. This allows developers who have contributed a package to be notified when a patch is proposed for the same package, and get the chance to review/test the patch.
  • Notify the developers of a package of build failures caused by this package in the Buildroot autobuilder infrastructure. So far, all build results of this infrastructure were simply sent every day on the mailing list, which made it unpractical for individual developers to notice which build failures they should look at. Thanks to the DEVELOPERS file, we now send a daily e-mail individually to the developers whose packages are affected by build failures.
  • Notify the developers of the support for a given CPU architecture of build failures caused on the autobuilders for this architecture, in a manner similar to what is done for packages.

Thanks to this, we have seen developers who were not regularly following the Buildroot mailing list contribute again fixes for build failures caused by their packages, increasing the overall Buildroot quality. The DEVELOPERS file and the get-developers script, as well as the related improvements to the autobuilder infrastructure were contributed by Thomas Petazzoni.

Build testing of defconfigs on Gitlab CI

Buildroot contains a number of example configurations called defconfigs for various hardware platforms, which allow to build a minimal embedded Linux system, known to work on such platforms. At the time of this writing, Buildroot has 146 defconfigs for platforms ranging from popular development boards (Raspberry Pi, BeagleBone, etc.) to evaluation boards from SoC vendors and Qemu machine emulation.

In order to make sure all those defconfigs build properly, we used to have a job running on Travis CI, but we started to face limitations, especially in the maximum allowed build duration. Therefore, Arnout Vandecappelle migrated this on Gitlab CI and things have been running smoothly since then.

Buildroot 2017.02 released, Bootlin contributions

Buildroot LogoThe 2017.02 version of Buildroot has been released recently, and as usual Bootlin has been a significant contributor to this release. A total of 1369 commits have gone into this release, contributed by 110 different developers.

Before looking in more details at the contributions from Bootlin, let’s have a look at the main improvements provided by this release:

  • The big announcement is that 2017.02 is going to be a long term support release, maintained with security and other important fixes for one year. This will allow companies, users and projects that cannot upgrade at each Buildroot release to have a stable Buildroot version to work with, coming with regular updates for security and bug fixes. A few fixes have already been collected in the 2017.02.x branch, and regular point releases will be published.
  • Several improvements have been made to support reproducible builds, i.e the capability of having two builds of the same configuration provide the exact same bit-to-bit output. These are not enough to provide reproducible builds yet, but they are a piece of the puzzle, and more patches are pending for the next releases to move forward on this topic.
  • A package infrastructure for packages using the waf build system has been added. Seven packages in Buildroot are using this infrastructure currently.
  • Support for the OpenRISC architecture has been added, as well as improvements to the support of ARM64 (selection of ARM64 cores, possibility of building an ARM 32-bit system optimized for an ARM64 core).
  • The external toolchain infrastructure, which was all implemented in a single very complicated package, has been split into one package per supported toolchain and a common infrastructure. This makes it much easier to maintain.
  • A number of updates has been made to the toolchain components and capabilities: uClibc-ng bumped to 1.0.22 and enabled for ARM64, mips32r6 and mips64r6, gdb 7.12.1 added and switched to gdb 7.11 as the default, Linaro toolchains updated to 2016.11, ARC toolchain components updated to arc-2016.09, MIPS Codescape toolchains bumped to 2016.05-06, CodeSourcery AMD64 and NIOS2 toolchains bumped.
  • Eight new defconfigs for various hardware platforms have been added, including defconfigs for the NIOSII and OpenRISC Qemu emulation.
  • Sixty new packages have been added, and countless other packages have been updated or fixed.
Buildroot developers at work during the Buildroot Developers meeting in February 2017, after the FOSDEM conference in Brussels.

More specifically, the contributions from Bootlin have been:

  • Thomas Petazzoni has handled the release of the first release candidate, 2017.02-rc1, and merged 742 patches out of the 1369 commits merged in this release.
  • Thomas contributed the initial work for the external toolchain infrastructure rework, which has been taken over by Romain Naour and finally merged thanks to Romain’s work.
  • Thomas contributed the rework of the ARM64 architecture description, to allow building an ARM 32-bit system optimized for a 64-bit core, and to allow selecting specific ARM64 cores.
  • Thomas contributed the raspberrypi-usbboot package, which packages a host tool that allows to boot a RaspberryPi system over USB.
  • Thomas fixed a large number of build issues found by the project autobuilders, contributing 41 patches to this effect.
  • Mylène Josserand contributed a patch to the X.org server package, fixing an issue with the i.MX6 OpenGL acceleration.
  • Gustavo Zacarias contributed a few fixes on various packages.

In addition, Bootlin sponsored the participation of Thomas to the Buildroot Developers meeting that took place after the FOSDEM conference in Brussels, early February. A report of this meeting is available on the eLinux Wiki.

The details of Bootlin contributions:

Buildroot 2016.11 released, Bootlin contributions

Buildroot LogoThe 2016.11 release of Buildroot has been published on November, 30th. The release announcement, by Buildroot maintainer Peter Korsgaard, gives numerous details about the new features and updates brought by this release. This new release provides support for using multiple BR2_EXTERNAL directories, gives some important updates to the toolchain support, adds default configurations for 9 new hardware platforms, and 38 new packages were added.

On a total of 1423 commits made for this release, Bootlin contributed a total of 253 commits:

$ git shortlog -sn --author=free-electrons 2016.08..2016.11
   142  Gustavo Zacarias
   104  Thomas Petazzoni
     7  Romain Perier

Here are the most important contributions we did:

  • Romain Perier contributed a package for the AMD Catalyst proprietary driver. Such drivers are usually not trivial to integrate, so having a ready-to-use package in Buildroot will really make it easier for Buildroot users who use hardware with an AMD/ATI graphics controller. This package provides both the X.org driver and the OpenGL implementation. This work was sponsored by one of Bootlin customer.
  • Gustavo Zacarias mainly contributed a large set of patches that do a small update to numerous packages, to make sure the proper environment variables are passed. This is a preparation change to bring top-level parallel build in Buildroot. This work was also sponsored by another Bootlin customer.
  • Thomas Petazzoni did contributions in various areas:
    • Added a DEVELOPERS file to the tree, to reference which developers are interested by which architectures and packages. Not only it allows the developers to be Cc’ed when patches are sent on the mailing list (like the get_maintainers script does), but it also used by Buildroot autobuilder infrastructure: if a package fails to build, the corresponding developer is notified by e-mail.
    • Misc updates to the toolchain support: switch to gcc 5.x by default, addition of gcc patches needed to fix various issues, etc.
    • Numerous fixes for build issues detected by Buildroot autobuilders

In addition to contributing 104 commits, Thomas Petazzoni also merged 1095 patches from other developers during this cycle, in order to help Buildroot maintainer Peter Korsgaard.

Finally, Bootlin also sponsored the Buildroot project, by funding the meeting location for the previous Buildroot Developers meeting, which took place in October in Berlin, after the Embedded Linux Conference. See the Buildroot sponsors page, and also the report from this meeting. The next Buildroot meeting will take place after the FOSDEM conference in Brussels.

Buildroot training updated to Buildroot 2016.05

Buildroot LogoAlmost exactly one year ago, we announced the availability of our training course on Buildroot. This course received very good feedback, both from our customers, and from the community.

In our effort to continuously improve and update our training materials, we have recently updated our Buildroot training course to Buildroot 2016.05, which was released at the end of May. In addition to adapting our practical labs to use this new Buildroot version, we have also improved the training materials to cover some of the new features that have been added over the last year in Buildroot. The most important changes are:

  • Cover the graph-size functionality, which allows to generate a pie chart of the filesystem size, per package. This is a very nice feature to analyze the size of your root filesystem and see how to reduce it.
  • Improve the description about the local site method and the override source directory functionalities, that are very useful when doing active application/library development in Buildroot, or to package custom application/library code.
  • Add explanations about using genimage to create complete SD card images that are ready to be flashed.
  • Add explanations about the hash file that can be added to packages to verify the integrity of the source code that is downloaded before it gets built.

The updated training materials are available on the training page: agenda (PDF), slides (PDF) and practical labs (PDF).

Contact us if you would like to organize this training session in your company: we are available to deliver it worldwide.

Factory flashing with U-Boot and fastboot on Freescale i.MX6

Introduction

For one of our customers building a product based on i.MX6 with a fairly low-volume, we had to design a mechanism to perform the factory flashing of each product. The goal is to be able to take a freshly produced device from the state of a brick to a state where it has a working embedded Linux system flashed on it. This specific product is using an eMMC as its main storage, and our solution only needs a USB connection with the platform, which makes it a lot simpler than solutions based on network (TFTP, NFS, etc.).

In order to achieve this goal, we have combined the imx-usb-loader tool with the fastboot support in U-Boot and some scripting. Thanks to this combination of a tool, running a single script is sufficient to perform the factory flashing, or even restore an already flashed device back to a known state.

The overall flow of our solution, executed by a shell script, is:

  1. imx-usb-loader pushes over USB a U-Boot bootloader into the i.MX6 RAM, and runs it;
  2. This U-Boot automatically enters fastboot mode;
  3. Using the fastboot protocol and its support in U-Boot, we send and flash each part of the system: partition table, bootloader, bootloader environment and root filesystem (which contains the kernel image).
The SECO uQ7 i.MX6 platform used for our project.
The SECO uQ7 i.MX6 platform used for our project.

imx-usb-loader

imx-usb-loader is a tool written by Boundary Devices that leverages the Serial Download Procotol (SDP) available in Freescale i.MX5/i.MX6 processors. Implemented in the ROM code of the Freescale SoCs, this protocol allows to send some code over USB or UART to a Freescale processor, even on a platform that has nothing flashed (no bootloader, no operating system). It is therefore a very handy tool to recover i.MX6 platforms, or as an initial step for factory flashing: you can send a U-Boot image over USB and have it run on your platform.

This tool already existed, we only created a package for it in the Buildroot build system, since Buildroot is used for this particular project.

Fastboot

Fastboot is a protocol originally created for Android, which is used primarily to modify the flash filesystem via a USB connection from a host computer. Most Android systems run a bootloader that implements the fastboot protocol, and therefore can be reflashed from a host computer running the corresponding fastboot tool. It sounded like a good candidate for the second step of our factory flashing process, to actually flash the different parts of our system.

Setting up fastboot on the device side

The well known U-Boot bootloader has limited support for this protocol:

The fastboot documentation in U-Boot can be found in the source code, in the doc/README.android-fastboot file. A description of the available fastboot options in U-Boot can be found in this documentation as well as examples. This gives us the device side of the protocol.

In order to make fastboot work in U-Boot, we modified the board configuration file to add the following configuration options:

#define CONFIG_CMD_FASTBOOT
#define CONFIG_USB_FASTBOOT_BUF_ADDR       CONFIG_SYS_LOAD_ADDR
#define CONFIG_USB_FASTBOOT_BUF_SIZE          0x10000000
#define CONFIG_FASTBOOT_FLASH
#define CONFIG_FASTBOOT_FLASH_MMC_DEV    0

Other options have to be selected, depending on the platform to fullfil the fastboot dependencies, such as USB Gadget support, GPT partition support, partitions UUID support or the USB download gadget. They aren’t explicitly defined anywhere, but have to be enabled for the build to succeed.

You can find the patch enabling fastboot on the Seco MX6Q uQ7 here: 0002-secomx6quq7-enable-fastboot.patch.

U-Boot enters the fastboot mode on demand: it has to be explicitly started from the U-Boot command line:

U-Boot> fastboot

From now on, U-Boot waits over USB for the host computer to send fastboot commands.

Using fastboot on the host computer side

Fastboot needs a user-space program on the host computer side to talk to the board. This tool can be found in the Android SDK and is often available through packages in many Linux distributions. However, to make things easier and like we did for imx-usb-loader, we sent a patch to add the Android tools such as fastboot and adb to the Buildroot build system. As of this writing, our patch is still waiting to be applied by the Buildroot maintainers.

Thanks to this, we can use the fastboot tool to list the available fastboot devices connected:

# fastboot devices

Flashing eMMC partitions

For its flashing feature, fastboot identifies the different parts of the system by names. U-Boot maps those names to the name of GPT partitions, so your eMMC normally requires to be partitioned using a GPT partition table and not an old MBR partition table. For example, provided your eMMC has a GPT partition called rootfs, you can do:

# fastboot flash rootfs rootfs.ext4

To reflash the contents of the rootfs partition with the rootfs.ext4 image.

However, while using GPT partitioning is fine in most cases, i.MX6 has a constraint that the bootloader needs to be at a specific location on the eMMC that conflicts with the location of the GPT partition table.

To work around this problem, we patched U-Boot to allow the fastboot flash command to use an absolute offset in the eMMC instead of a partition name. Instead of displaying an error if a partition does not exists, fastboot tries to use the name as an absolute offset. This allowed us to use MBR partitions and to flash at defined offset our images, including U-Boot. For example, to flash U-Boot, we use:

# fastboot flash 0x400 u-boot.imx

The patch adding this work around in U-Boot can be found at 0001-fastboot-allow-to-flash-at-a-given-address.patch. We are working on implementing a better solution that can potentially be accepted upstream.

Automatically starting fastboot

The fastboot command must be explicitly called from the U-Boot prompt in order to enter fastboot mode. This is an issue for our use case, because the flashing process can’t be fully automated and required a human interaction. Using imx-usb-loader, we want to send a U-Boot image that automatically enters fastmode mode.

To achieve this, we modified the U-Boot configuration, to start the fastboot command at boot time:

#define CONFIG_BOOTCOMMAND "fastboot"
#define CONFIG_BOOTDELAY 0

Of course, this configuration is only used for the U-Boot sent using imx-usb-loader. The final U-Boot flashed on the device will not have the same configuration. To distinguish the two images, we named the U-Boot image dedicated to fastboot uboot_DO_NOT_TOUCH.

Putting it all together

We wrote a shell script to automatically launch the modified U-Boot image on the board, and then flash the different images on the eMMC (U-Boot and the root filesystem). We also added an option to flash an MBR partition table as well as flashing a zeroed file to wipe the U-Boot environment. In our project, Buildroot is being used, so our tool makes some assumptions about the location of the tools and image files.

Our script can be found here: flash.sh. To flash the entire system:

# ./flash.sh -a

To flash only certain parts, like the bootloader:

# ./flash.sh -b 

By default, our script expects the Buildroot output directory to be in buildroot/output, but this can be overridden using the BUILDROOT environment variable.

Conclusion

By assembling existing tools and mechanisms, we have been able to quickly create a factory flashing process for i.MX6 platforms that is really simple and efficient. It is worth mentioning that we have re-used the same idea for the factory flashing process of the C.H.I.P computer. On the C.H.I.P, instead of using imx-usb-loader, we have used FEL based booting: the C.H.I.P indeed uses an Allwinner ARM processor, providing a different recovery mechanism than the one available on i.MX6.