Bootlin will be preent at Embedded World 2020, in Nuremberg on February 25-27. We will be present on STMicroelectronics booth in hall 4A, stand 138. We will have two demos of the STM32MP1 platform running Linux, and of course details about Bootlin services around embedded Linux and Linux kernel development and training.
Three people from Bootlin will be present: Michael Opdenacker (CEO), Thomas Petazzoni (CTO) and Alexandre Belloni (embedded Linux engineer and trainer).
Do not hesitate to get in touch with us prior to the event if you would like to schedule a meeting to discuss business, project or career opportunities.
According to the statistics, a total of 14350 changes were made to this kernel release, to which Bootlin contributed 124 patches, making us the 19th contributing company by number of commits. Here are the highlights of our contributions:
By far and large our most important achievement in Linux 5.5 is the merge of the H265 decoding support in the Allwinner VPU driver, developed by Paul Kocialkowski. This was the last missing feature to complete the effort funded by the Kickstarter campaign we launched in February 2018. See our blog post wrapping up the Allwinner VPU work.
Alexandre Belloni as the RTC subsystem maintainer, as usual contributed a large number of RTC driver improvements and fixes.
Antoine Ténart contributed some improvements to the Cadence MACB driver, most notably used as the Ethernet controller driver on Microchip/Atmel platforms. The main improvement is the conversion to the phylink subsystem for the interaction with the Ethernet PHY.
Grégory Clement contributed numerous enhancements to the Microchip/Atmel SPI controller driver, mainly aimed at fixing and improving the support for Chip Select, both native Chip Selects and GPIO-based Chip Selects.
Kamel Bouhara contributed a few additional Device Tree files to describe home automation hardware platforms from Overkiz, based on the Microchip SAMA5D2 processor, as well as an improvement to a Microchip SoC driver that allows to expose the platform serial number to user-space.
Miquèl Raynal added support for a new Marvell system-on-chip called Marvell CN9130. Despite the name, this chip is from a hardware point of view in the same family as the Marvell Armada 7K and 8K, which were already supported upstream.
Miquèl Raynal contributed a number of fixes and improvements to the Macronix SPI controller driver.
Miquèl Raynal added support for the ADC converters over SPI MAX1227, MAX1229 and MAX1231 to the existing max1027 IIO driver in the kernel. These ADCs have a 12-bit resolution.
Miquèl Raynal improved the SPI controller driver for the Zynq system-on-chips to correctly support multiple Chip Selects.
In addition to these direct contributions, some of the Bootlin engineers are also Linux kernel maintainers and therefore review and merge patches from other contributors: Alexandre Belloni as the RTC subsystem maintainer and Microchip platform co-maintainer reviewed and merged 45 patches from other contributors, Miquèl Raynal as the MTD subsystem co-maintainer reviewed and merged 39 patches from other contributors and Grégory Clement as the Marvell platform co-maintainer reviewed and merged 29 patches from other contributors.
This week-end takes place one of the biggest and most important free and open-source software conference in Europe: FOSDEM. It will once again feature a very large number of talks, organized in several main tracks and developer rooms.
Bootlin CTO Thomas Petazzoni will participate to the FOSDEM conference, of course attending many of the talks from the Embedded, Mobile and Automative Devroom, to which he participated to the talk review and selection. Do not hesitate to get in touch with Thomas if you want to discuss career or business opportunities with Bootlin.
In addition, Thomas will also participate to the 3-day Buildroot Developers meeting which takes place in Brussels right after the FOSDEM conference, kindly hosted by Google. During 3 days, some of the core Buildroot developers will work together to discuss the future of Buildroot, as well as review and discuss pending patches and proposals.
So far, we have used a microSD card as storage for the Linux system running on the STM32MP1 platform. Since this media is removable, we can easily switch the microSD card back and forth between the STM32MP1 platform and our development workstation, which is nice during development and debugging.
However, an actual product will most likely use some form of non-removable persistent storage, typically an eMMC or a NAND flash. While not available on the STM32MP1-DK1 board probably for cost reasons, these storage devices are very common in most embedded systems. For example, the STM32MP157A-EV1 board provides three non-removable persistent storage devices: a 4 GB eMMC, a 1 GB NAND flash, and a 64 MB QSPI NOR flash.
When such storage devices are shipped by their manufacturer, they are typically empty. Therefore, as part of the manufacturing process of your embedded systems, you will have to load the relevant storage device with your Linux system, applications and data, so that the embedded system is fully operational: this is the process referred to as factory flashing in this blog post.
If you are doing a very high volume product, you can ask your eMMC or NAND flash vendor to pre-load a system image on the storage before it is shipped to you and assembled on your board. However, many companies do products with volumes that are not large enough to make such a strategy possible: in this case, you really receive an empty storage device, and have to flash it.
A first possibility to flash the non-removable storage is to use a removable storage device, boot a Linux system on the device, and use it to flash the non-removable storage. This can definitely be a possible option in some situations, but it is not always possible (if there’s no removable storage device interface at all) or not always practical.
However, most system-on-chips, including the STM32MP1 include some ROM code that the processor executes at boot time, even before it loads the first stage bootloader. This ROM code is primarly responsible for loading the first stage bootloader into memory, but it also very often offers a communication channel with the outside world, which can be used to gain control of a platform that has nothing at all on its storage. This communication channel is typically over USB or UART, and most often uses a custom, vendor-specific protocol, which is understood by vendor-specific tools. This protocol generally allows to send some code to the target and get it executed, which is sufficient to be able to reflash the target device.
Here are a few examples with system-on-chips from various vendors:
The ROM code of the NXP i.MX processors implements a USB-based protocol, which can be interfaced either using the NXP-provided mfgtools, or using the community-developed imx_usb_loader. The latter was presented in one of our earlier blog posts about i.MX6 factory flashing.
The ROM code of Microchip SAMA5 processors implements a USB-based protocol, which can be interfaced using a tool called SAM-BA
The ROM code of Rockchip processors implements a USB-based protocol, which can be interfaced either using a Rockchip-specific tool called rkdeveloptool
The ROM code of the ST STM32MP15 processors also implement a USB-based protocol, which can be interfaced using the STM32 Cube Programmer
Obviously, in this blog post, we are going to use the latter, STM32 Cube Programmer, to flash our STM32MP1 platform. Since the DK2 board only has a removable device, we will use the tool to flash the SD card, but the process and logic would be the same for any other (non-removable) storage device.
Getting and installing STM32 Cube Programmer
While ST generally has very good upstream and open-source support for its products, the STM32 Cube Programmer unfortunately doesn’t follow this strategy: you need to be registered on the ST web site to download it, and its source code is not available. Due to this registration process, we for example cannot create a Buildroot package that would automatically download and install this tool for you.
So, follow the process to create an account on the ST web site, and then go to the STM32 Cube Programmer page. At the time of this writing, the latest version is 2.2.1, but according this Wiki page, this version doesn’t work for the STM32MP1 platform. Instead, select to download the 2.2.0 version, which is known to work. You will then download a file called en.stm32cubeprog.zip (it doesn’t have the version in its name, which isn’t great) weighting 187 MB, and which has the SHA256 hash 91107b4d605d126f5c32977247d7419d42abb2655848d2d1a16d52f5f7633d2d.
Extract this ZIP file somewhere in your system, and then run the SetupSTM32CubeProgrammer-2.2.0.linux executable:
$ ./SetupSTM32CubeProgrammer-2.2.0.linux
Got through the installation steps. On our system, we customized the installation path to be just $HOME/stm32cube, and the remainder of this blog post will assume this is where you installed the STM32 Cube Programmer.
In this blog post, we are only going to use the command line interface (CLI) of STM32 Cube Programmer, so just make sure you can run the corresponding tool:
On the back of the board, there is a two-way DIP switch labeled SW1, which is used to configure the boot mode. When both are “ON”, the board boots from the SD card. When both are “OFF”, the board enters the “USB boot for flashing mode”, which is what we are going to use. So switch both switches to OFF, and reset the board.
Plug an additional USB-C cable from the board CN7 connector (which is located between the HDMI port and the 4 USB host ports).
Then, reset the board. If you run lsusb on your Linux workstation, you should see a new device:
Bus 003 Device 011: ID 0483:df11 STMicroelectronics STM Device in DFU Mode
Then, you can ask STM32_Programmer_CLI to list the devices it sees over USB. This needs root permissions (unless appropriate udev rules are created):
$ sudo ~/stm32cube/bin/STM32_Programmer_CLI -l usb
-------------------------------------------------------------------
STM32CubeProgrammer v2.2.0
-------------------------------------------------------------------
===== DFU Interface =====
Total number of available STM32 device in DFU mode: 1
Device Index : USB1
USB Bus Number : 003
USB Address Number : 003
Product ID : DFU in HS Mode @Device ID /0x500, @Revision ID /0x0000
Serial number : 004200343338510534383330
Firmware version : 0x0110
Device ID : 0x0500
Good, the STM32CubeProgrammer tool is seeing our board, and we see that the Device Index is USB1. Keep that in mind for the next steps.
Change the Linux system boot chain
STM32CubeProgrammer works by sending a U-Boot bootloader over USB, and then talking to this U-Boot to make it erase the MMC or NAND flash, and make it write some data to those storage devices. However, for some reason, STM32CubeProgrammer doesn’t work with the boot flow we have used so far, which uses the U-Boot SPL as the first-stage bootloader, and U-Boot itself as the second stage bootloader. It only works when the first stage bootloader is the Arm Trusted Firmware, also called TF-A. You can get more details about the different possible boot chains on STM32MP1 on this Wiki page.
Due to this constraint, we are going to switch our Buildroot configuration to use TF-A instead of U-Boot SPL as the first stage bootloader.
First of all, we need to backport two Buildroot commits, which did not exist in the Buildroot 2019.02 we are using, but have been integrated later. The first commit, 9dbc934217e170578d4cbfdf524bc1b3988d0b9e allows to build TF-A for ARM 32-bit platforms, while the second commit, e4d276c357fdf9f19f99f826cab63f373687f902 allows to provide a custom name for the TF-A image name.
The second one will cause some minor conflict in boot/arm-trusted-firmware/Config.in. Resolve the conflict by removing the BR2_TARGET_ARM_TRUSTED_FIRMWARE_DEBUG option from this file, remove the conflict markers, then run:
If you’re not sure about this, you can check our 2019.02/stm32mp157-dk-blog-6 branch on Github, which has these changes already integrated.
Once done, we can run make menuconfig and start modifying the Buildroot configuration. Here are the changes that we need:
In the Bootloaders menu, enable ARM Trusted Firmware (ATF), and then:
Set ATF Version to Custom Git repository
Set URL of custom repository to https://github.com/STMicroelectronics/arm-trusted-firmware.git
Set Custom repository version to v2.0-stm32mp-r2
Set ATF platform to stm32mp1
Set Additional ATF build variables to DTB_FILE_NAME=stm32mp157c-dk2.dtb AARCH32_SP=sp_min. The DTB_FILE_NAME selects the correct Device Tree file for the DK2 board, while the AARCH32_SP indicates that we are using the “minimal” secure payload, and not a complete Trusted Execution Environment such as OP-TEE.
Set Binary boot images to *.stm32. This makes sure the final image gets copied to output/images.
Still in the Bootloaders menu, inside the U-Boot option, make the following changes:
Change Board defconfig to stm32mp15_trusted. This is the most important change, which makes U-Boot build only the second stage, and in a format that gets loaded by TF-A as the first stage.
In U-Boot binary format, disable u-boot.img, and instead enable Custom (specify below) and indicate u-boot.stm32 as the value for U-Boot binary format: custom names.
Disable the Install U-Boot SPL binary image option.
Overall, the diff of the changes in the configuration looks like this:
@@ -30,16 +30,22 @@ BR2_TARGET_ROOTFS_EXT2=y
BR2_TARGET_ROOTFS_EXT2_4=y
BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
# BR2_TARGET_ROOTFS_TAR is not set
+BR2_TARGET_ARM_TRUSTED_FIRMWARE=y
+BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_GIT=y
+BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_REPO_URL="https://github.com/STMicroelectronics/arm-trusted-firmware.git"
+BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_REPO_VERSION="69cc28c5a1b877cf67def7f94dece087f3917b1c"
+BR2_TARGET_ARM_TRUSTED_FIRMWARE_PLATFORM="stm32mp1"
+BR2_TARGET_ARM_TRUSTED_FIRMWARE_ADDITIONAL_VARIABLES="DTB_FILE_NAME=stm32mp157c-dk2.dtb AARCH32_SP=sp_min"
+BR2_TARGET_ARM_TRUSTED_FIRMWARE_IMAGES="*.stm32"
BR2_TARGET_UBOOT=y
BR2_TARGET_UBOOT_BUILD_SYSTEM_KCONFIG=y
BR2_TARGET_UBOOT_CUSTOM_GIT=y
BR2_TARGET_UBOOT_CUSTOM_REPO_URL="https://github.com/STMicroelectronics/u-boot.git"
BR2_TARGET_UBOOT_CUSTOM_REPO_VERSION="v2018.11-stm32mp-r2.1"
-BR2_TARGET_UBOOT_BOARD_DEFCONFIG="stm32mp15_basic"
+BR2_TARGET_UBOOT_BOARD_DEFCONFIG="stm32mp15_trusted"
BR2_TARGET_UBOOT_CONFIG_FRAGMENT_FILES="board/stmicroelectronics/stm32mp157-dk/uboot-fragment.config"
# BR2_TARGET_UBOOT_FORMAT_BIN is not set
-BR2_TARGET_UBOOT_FORMAT_IMG=y
-BR2_TARGET_UBOOT_SPL=y
-BR2_TARGET_UBOOT_SPL_NAME="spl/u-boot-spl.stm32"
+BR2_TARGET_UBOOT_FORMAT_CUSTOM=y
+BR2_TARGET_UBOOT_FORMAT_CUSTOM_NAME="u-boot.stm32"
BR2_TARGET_UBOOT_CUSTOM_MAKEOPTS="DEVICE_TREE=stm32mp157c-dk2"
BR2_PACKAGE_HOST_GENIMAGE=y
Before we can restart the build, we need to adjust the genimage.cfg file that describes the layout of the SD card. Indeed, the file name of the first stage bootloader is now tf-a-stm32mp157c-dk2.stm32 instead of u-boot-spl.stm32 and the file name of the second stage bootloader is now u-boot.stm32 instead of u-boot.img. All in all, your genimage.cfg file in board/stmicroelectronics/stm32mp157-dk/genimage.cfg should now look like this:
With this in place, it’s time to restart the build. You can do a complete rebuild with make clean all, or you can just clean up U-Boot, and restart the build:
$ make uboot-dirclean
$ make
You should now have in output/images the new TF-A image tf-a-stm32mp157c-dk2.stm32 and the new U-Boot image u-boot.stm32. Of course the sdcard.img file has been updated.
Rather than updating our SD card on our workstation, we’ll directly use the STM32CubeProgrammer tool to do that, in the next section.
Flashing the board
The STM32CubeProgrammer tool takes as input a flash layout file, which has a .tsv extension. The format of this file is extensively documented on this Wiki page. It is essentially a text file that says what should be flashed in each partition.
In our case, we are going to simply flash the entire sdcard.img instead of flashing partition by partition. To achieve this, we are going to use the RawImage image type, also described on the Wiki page.
Let’s create a file board/stmicroelectronics/stm32mp157-dk/flash.tsv, with the following contents:
#Opt Id Name Type IP Offset Binary
- 0x01 fsbl1-boot Binary none 0x0 tf-a-stm32mp157c-dk2.stm32
- 0x03 ssbl-boot Binary none 0x0 u-boot.stm32
P 0x10 sdcard RawImage mmc0 0x0 sdcard.img
The first line is a comment, just to help remember what each field is about. The second and third lines tell STM32CubeProgrammer which bootloader images should be used as part of the flashing process. Finally, the last line says we want to flash sdcard.img as a raw image on the mmc0 device.
Then, go do output/images, and run STM32CubeProgrammer. We use the -c port=usb1 argument, because our board was detected as device USB1 when we enumerated all detected devices using the -l usb previously.
$ cd output/images/
$ sudo ~/stm32cube/bin/STM32_Programmer_CLI -c port=usb1 -w ../../board/stmicroelectronics/stm32mp157-dk/flash.tsv
The output will look like this:
-------------------------------------------------------------------
STM32CubeProgrammer v2.2.0
-------------------------------------------------------------------
USB speed : High Speed (480MBit/s)
Manuf. ID : STMicroelectronics
Product ID : DFU in HS Mode @Device ID /0x500, @Revision ID /0x0000
SN : 004200343338510534383330
FW version : 0x0110
Device ID : 0x0500
Device name : STM32MPxxx
Device type : MPU
Device CPU : Cortex-A7
Start Embedded Flashing service
Memory Programming ...
Opening and parsing file: tf-a-stm32mp157c-dk2.stm32
File : tf-a-stm32mp157c-dk2.stm32
Size : 237161 Bytes
Partition ID : 0x01
Download in Progress:
[==================================================] 100%
File download complete
Time elapsed during download operation: 00:00:00.444
RUNNING Program ...
PartID: :0x01
Start operation done successfully at partition 0x01
Flashlayout Programming ...
[==================================================] 100%
Running Flashlayout Partition ...
Flashlayout partition started successfully
Memory Programming ...
Opening and parsing file: u-boot.stm32
File : u-boot.stm32
Size : 748042 Bytes
Partition ID : 0x03
Download in Progress:
[==================================================] 100%
File download complete
Time elapsed during download operation: 00:00:00.791
RUNNING Program ...
PartID: :0x03
reconnecting the device ...
USB speed : High Speed (480MBit/s)
Manuf. ID : STMicroelectronics
Product ID : USB download gadget@Device ID /0x500, @Revision ID /0x0000
SN : 004200343338510534383330
FW version : 0x0110
Device ID : 0x0500
Start operation done successfully at partition 0x03
Memory Programming ...
Opening and parsing file: sdcard.img
File : sdcard.img
Size : 539002368 Bytes
Partition ID : 0x10
Download in Progress:
[==================================================] 100%
File download complete
Time elapsed during download operation: 00:04:31.583
RUNNING Program ...
PartID: :0x10
Start operation done successfully at partition 0x10
Flashing service completed successfully
Finally, we can toggle back the SW1 DIP switches to their ON position, to boot again from SD card, and hit the reset button. The board should boot, but this time with our new image, which uses TF-A instead of U-Boot SPL, so the first lines of the boot process should look like this:
In this article, we have discussed the concept of factory flashing, understood better the different boot chains available for the STM32MP1, switched to a boot chain using TF-A, and presented how to use STM32CubeProgrammer to reflash the entire SD card.
As usual, we have a branch on Github with the Buildroot changes corresponding to this blog post, see the branch 2019.02/stm32mp157-dk-blog-6.
Stay tuned for the next article in this series of blog post, in which we will cover the topic of Over-The-Air firmware update.
First of all, the entire team at Bootlin wishes you a Happy New Year, and best wishes for 2020 in your personal and professional life. The beginning of the new year is a good time to look back and see the achievements of the past year, which is why we review the 2019 year in terms of Bootlin news and activity.
The brand new subsystem to support the MIPI I3C bus that we developed from scratch was merged in Linux 5.0. This paved the way to support both I3C controllers and I3C devices in Linux. See our blog post.
The Marvell Ethernet controller driver was extended to support packet classification offloading in hardware.
A brand new driver in the IIO subsystem for the TI ADS8344 ADC chip.
Support for HW-accelerated H264 video decoding was added to the Allwinner VPU driver, as part of our crowd-funded project, see our recent blog post.
Numerous improvements in the support of Allwinner, Microchip and Marvell platforms, see our blog posts for each kernel release for more details.
Other contributions
We contributed 52 patches to the U-Boot project in 2019: a new network driver the Microsemi Ocelot platform, improvements to the MTD subsystem, fixes to the NXP LPC3250 and Rockchip PX30 platform support.
We contributed 314 patches to the Buildroot project, our most significant contribution is support for top-level parallel build, which will land in the upcoming Buildroot 2020.02 release. In addition, Bootlin engineer Thomas Petazzoni remains an active Buildroot co-maintainer: in 2019 he reviewed and merged 2924 patches from other contributors, out of the total of 5503 patches merged throughout the year.
We improved our Web-based Elixir code browser. Elixir now indexes the sources of 16 projects, allows to browse various types of include files, has support for project specific HTML post-processing filters, has a new REST API (thanks to Carmeli Tamir), and has many fixed bugs. All this activity corresponds to a round number of commits: 128.
Engineering projects
Of course, most of the contributions described above are driven by the engineering projects we have with our customers worldwide. Here are some of the significant engineering projects we worked on in 2019:
Implemented support for MACsec hardware offloading, and support for this functionality for the Microchip VSC8584 Ethernet PHY. We presented this work in detail in a blog post, and submitted 4 iterations in 2019 and hope to see this merged in early 2020.
Improved the Intel GMA500 display driver to support page flipping, so that one of our customers can use the Weston Wayland compositor on hardware platforms that use this display controller.
Started a project to support the Microchip VSC8572 Ethernet PHY in Linux, used as a pass-through to SFP cages. As part of this project, we already contributed patches to convert the cpsw network driver to phylink, and we will contribute VSC8572 patches once the project has made enough progress.
Migrated a complete Buildroot-based BSP for a Danish customer in the healthcare industry: migration to a newer Linux kernel version and a newer Buildroot version for a Microchip AT91SAM9G45 platform and a TI AM335x platform.
Migrated a complete Yocto-based BSP for a Belgian customer to a newer mainline version of the Linux kernel for an i.MX6 platform, implemented secure boot and optimized the boot time.
For a major US customer, implemented a complete Linux BSP for a custom Xilinx Zynq 7000 platform: upstream U-Boot, upstream Linux and Yocto-based build system. As part of this project, we did a number of kernel contributions and we still have a major kernel contribution pending: a complete DRM driver for the logiCVC display controller.
For a German customer in the healthcare industry, continued to support additional hardware features of an i.MX6 platform, and built a complete new BSP for an SAMA5D3-based hardware platform.
For a Canadian customer, completely upgraded the Linux BSP for a NXP LPC3250 platform: update to U-Boot upstream, to Linux upstream, and migration to Buildroot as a build system.
For an Italian customer, started a brand new BSP for a Rockchip PX30 based platform, which will require a number of improvements and additions to the Linux kernel support for this platform, which we will work on in 2020.
For a Belgian customer, migrated a Linux BSP for an OMAP44xx platform with complex audio interfaces to a recent upstream Linux kernel version.
For a French customer, implemented and delivered a complete Yocto-based Linux BSP for an i.MX6 platform.
For the French company Overkiz, updated, cleaned-up and upstreamed to the Linux kernel the Device Tree files for their Microchip SAMA5 home automation platforms.
Implemented support for software-based ECC and external hardware ECC engines in the SPI flash subsystem of the Linux kernel, for one of our customers that manufactures flash chips. This patch series is still under review, but we expect to get it merged in early 2020.
Implemented support for dm-verity and SELinux in OpenWRT, for one of our customers. See our blog post.
Continued to work on implementing top-level parallel build in Buildroot, which finally got merged at the end of 2019 in upstream Buildroot.
Continued to work on improving the support for Microchip ARM processors and Marvell Armada ARM processors in the Linux kernel, in many areas.
For a customer building a complex audio product based on an Allwinner system-on-chip, we implemented support for TDM in the Allwinner audio interface driver and used it in conjunction with PDM microphones.
Published a long series of blog posts on how to create a Linux system with Buildroot for the STM32MP1 platform: part 1, part 2, part 3, part 4 and part 5. This series will continue in 2020.
Completed the work on the Allwinner VPU that was funded by the Kickstarter campaign of early 2018: the Allwinner VPU driver was merged, with MPEG2, H264 and H265 decoding support, see our blog post.
Continued to work on the support for the RaspberryPi platforms, making a number of enhancements to the display support and its testing, as we reported in a blog post.
Training
Our training business has seen quite a bit of activity in 2019:
We created and published a new training course: Displaying and rendering graphics with Linux. This 2-day course, created by Bootlin engineer Paul Kocialkowski provides a detailed walk-through of Linux graphics: graphics hardware and theory, low-level support in the Linux kernel, support in user-space in Wayland or X.org, OpenGL acceleration, etc. This course is available on-site and we already delivered it to customers in Spain, Portugal and France.
Our Embedded Linux and Yocto/OpenEmbedded training courses have been ported to use the STM32MP1 platform for practical labs, and we became a member of the STMicroelectronics Partner Program. The support for STM32MP1 in our training courses is proposed as an alternative to the Microchip SAMA5D3 platform (for the Embedded Linux training) and the BeagleBone Black (for the Yocto training) that we continue to support as well.
Overall our publicly available training materials have received 376 commits during the course of 2019, all visible in their GitHub repository.
We delivered many of our on-site training courses in France, India, Spain, Serbia, Poland, Finland, Portugal, Nederlands and Austria, and continued to offer our training courses in public sessions in Avignon, France.
Conferences
As usual in 2019, we attended and participated to a number of conferences:
Our networking experts Antoine Ténart and Maxime Chevallier attended the Netdev 0x13 conference in Prague.
Bootlin engineer Alexandre Belloni attended the SiFive Tech Symposium about the RISC-V architecture, in Grenoble.
A significant part of our engineering team attended the Linux Plumbers conference in Lisbon.
Bootlin engineer Grégory Clement attended the Kernel Recipes conference in Paris.
An additional engineer joined our team in Lyon: Kamel Bouhara. Kamel is now working with our senior engineers Grégory Clement and Alexandre Belloni in this office.
During the summer 2019, Victor Huesca joined Bootlin as an intern in Toulouse, and worked on improving the tooling used for the maintenance of the Buildroot project. All the work done by Victor is now used by the Buildroot community, see our blog post for more details.
An additional engineer will join our Toulouse office at the end of January 2020, and 3 interns will also join Bootlin during the first half of 2020, working on improving the Elixir Cross Referencer, and contributing to Linux and U-Boot.
We still have positions opened for Embedded Linux and kernel engineers, see job offer.
Back in early 2018, Bootlin started a crowd-funding campaign to fund the development of an upstream Linux kernel driver for the VPU found in Allwinner processors. Thanks to the support from over 400 contributors, companies and individuals, we have been able to bring support for hardware-accelerated video decoding in the mainline Linux kernel for Allwinner platforms.
From April 2018 to end of 2019, Paul Kocialkowski and Maxime Ripard at Bootlin worked hard on developing the driver and getting it accepted upstream, as well as developing the corresponding user-space components. We regularly published the progress of our work on this blog.
As of the end of 2019, we can say that all the goals defined in the Kickstarter have been completed:
We have an upstream Linux kernel for the Allwinner VPU, in drivers/staging/media/sunxi/cedrus, which supports MPEG2 decoding (since Linux 4.19), H264 decoding (since Linux 5.2) and H265 decoding (will be in the upcoming Linux 5.5)
We have a user-space VA-API implementation called libva-v4l2-request, and which allows to use any Linux kernel video codec based on the request API.
We have enabled the Linux kernel driver on all platforms we listed in our Kickstarter campaign: A13/A10S/A20/A33/H3 (since Linux 4.19), A64/H5 (since Linux 4.20), A10 (since Linux 5.0) and H6 (since Linux 5.1, contributed by Jernej Skrabec)
This means that the effort that was funded by the Kickstarter campaign is now over, and from now on, we are operating in maintenance mode regarding the cedrus driver: we are currently not actively working on developing new features for the driver anymore.
The Cedrus Linux driver in action
Of course, there are plenty of additional features that can be added to the driver: support for H264 encoding, support for high-profile H264 decoding, support for other video codecs. Bootlin is obviously available to develop those additional features for customers, do not hesitate to contact us if you are interested.
Overall, we found this experience of funding upstream Linux kernel development through crowd-funding very interesting and we’re happy to have been successful at delivering what was promised in our campaign. Looking at the bigger picture, the Linux userspace API for video decoding with stateless hardware codecs in V4L2 has been maturing for a while and is getting closer and closer to being finalized and declared a stable kernel API: this project has been key in the introduction of this API, as cedrus was the first driver merged to require and use it. Additional drivers are appearing for other stateless decoding engines, such as the Hantro G1 (found in Rockchip, i.MX and Microchip platforms) or the rkvdec engine. We are of course also interested in working on support for these VPUs, as we have gained significant familiarity with all things related to hardware video decoding during the cedrus adventure.
This time around, we’re quite late to the party, but Linux 5.4 was indeed released a number of weeks ago, and once again, Bootlin contributed a number of patches to this Linux kernel release. As usual, the most useful source of information to learn about the major features brought by Linux 5.4 are the LWN articles (part 1, part 2) and the KernelNewbies Wiki.
With a total of 143 patches contributed to this release, Bootlin is the 17th contributing company by number of commits acccording to the Linux Kernel Patch Statistic.
Here are the highlights of our contributions:
Antoine Ténart contributed support for IEEE 1588 Precision Time Protocol (PTP) to the Microsemi Ocelot Ethernet switch driver, which Bootlin developed and upstreamed in 2018 (see our blog post)
In the MTD subsystem, a number of contributions to the spi-nor support, written originally by Boris Brezillon, made their way upstream.
In the support of Microchip (formerly Atmel) platforms, Kamel Bouhara, who joined Bootlin in September 2019, sees his first kernel contribution merged as a Bootlin engineer: dropping useless support for platform_data from the Atmel PWM driver.
In the support of Allwinner platforms
Maxime Ripard contributed a brand new driver for the Allwinner A10 camera interface driver, a driver that we started at Bootlin for the CHIP platform back in the days, and that we finished more recently.
Maxime Ripard contributed a significant number of improvements to the sun4i-i2saudio interface driver, especially TDM support, which was developed as part of a customer project at Bootlin.
Maxime Ripard also contributed numerous enhancements to Allwinner platform Device Tree files, especially in the area of using YAML schemas.
In the support for Marvell platforms
Grégory Clement added cpufreq support to the Marvell Armada 7K/8K platform by extending some of its clock drivers.
Miquèl Raynal contributed improvements to the Marvell CP110 COMPHY driver, which is used to control SERDES lanes on the Marvell Armada 7K/8K platforms, and added the description of the SERDES lanes used by various IP blocks in those processors.
Alexandre Belloni, as the RTC subsystem maintainer, did a number of fixes and improvements in several RTC drivers (mainly pcf2123 and pcf8563)
For the LPC3250 platform, for which Bootlin delivered a modern BSP to a customer last year, Alexandre Belloni fixed an issue in the lpc_eth network driver, which was preventing the system from booting if the network had been initialized by the bootloader.
In addition to being contributors, some Bootlin engineers are also maintainers of various parts of the Linux kernel, and as such review and merge code from other contributors:
As the RTC subsystem maintainer and Microchip platform co-maintainer, Alexandre Belloni merged 47 patches from other contributors
As the MTD subsystem co-maintainer, Miquèl Raynal merged 33 patches from other contributors
As the Marvell platform co-maintainer, Grégory Clement merged 11 patches from other contributors
Here are the details of all our contributions to Linux 5.4:
While Bootlin is largely known for its expertise with the Buildroot and Yocto/OpenEmbedded build systems, we do also work with other build systems for customer projects. Specifically, in 2019, we have worked for one of our customers on extending OpenWrt to add support for two security features: dm-verity and SELinux, which we have contributed to upstream OpenWrt. In this blog post, we provide some details about those features, and how they are integrated in OpenWrt.
dm-verity support
dm-verity is a device mapper target that allows to create a block device on top of an existing block device, with a transparent integrity checking in-between. Provided a tree of per-block hashes that is generated offline, dm-verity will verify at run-time that all the data read from the underlying block device matches the hashes that are provided. This allows to guarantee that the data has not been modified, as a root hash must be passed from a trusted source when setting up the dm-verity block device at boot time. If any bit in the storage has been modified, the verification of the hashes all the way up to the root hash will fail, and the I/O operation on the block of data being read from storage will be rejected. Therefore, dm-verity is typically used as part of a secure boot strategy, which allows the root hash to be passed by the bootloader to the kernel, where the bootloader and kernel themselves are verified by other means. Also, due to the nature of the integrity verification, dm-verity provides a read-only block device, and will therefore only work with read-only filesystems.
We implemented an integration of this mechanism in OpenWrt, and contributed a first version back in March, and we just sent a second version in November.
In essence, our integration consists in:
Packaging the different tools that are needed to generate at build time the tree of hashes corresponding to a given filesystem image. The important package here is cryptsetup (patch 05/12), but it requires packaging a few dependencies: libjson-c (patch 04/12), popt (patch 03/12), lvm2 (patch 02/12) and libaio (patch 01/12)
Extending the mechanism of OpenWrt to generate FIT images for the kernel so that it can include a U-Boot script (patch 06/12 and patch 08/12). Indeed, we have chosen to embed the root hash information inside the FIT image, as FIT image can be signed and verified by the bootloader before booting, ensuring that the root hash is part of the trusted information.
Extending the squashfs filesystem image generation logic so that a dm-verity-capable image can optionally be generated (patch 09/12). If this is the case, then the squashfs image itself is concatenated with the tree of hashes, and a U-Boot script containing the details of the dm-verity image is generated. This includes the important root hash information.
Backporting to the 4.14 and 4.19 Linux kernels currently supported by OpenWrt the DM_INIT mechanism that is in upstream Linux since 5.1, and which allows to setup a device mapper target at boot time using the dm-mod.create= kernel argument (patch 10/12). This allows to have the root filesystem on a device mapper block device, without the need for an initramfs to setup the device mapper target.
Showing with the example of the Marvell Armada XP GP platform how to enable this mechanism on a specific hardware platform already supported by OpenWrt (patch 11/12 and patch 12/12).
For more details, you can read the cover letter of the patch series.
SELinux support
SELinux is a Linux security module that implements Mandatory Access Control and that is generally pretty infamously known in the Linux user community for being difficult to use and configure. However, it is widely used in security-sensitive systems, including embedded systems and as such, makes sense to see supported in OpenWrt. For example, SELinux is already supported in the Yocto/OpenEmbedded ecosystem through the meta-selinux layer, and in the Buildroot project since 2014, contributed by Collins Aerospace.
In short, the basic principle of SELinux is that important objects in the system (files, processes, etc.) are associated to a security context. Then, a policy defines which operations are allowed, depending on the security context of who is doing the operation and on what the operation takes place. This policy is compiled into a binary policy, which is loaded into the kernel early at boot time, and then enforced by the kernel during the system life. Of course, around this, SELinux provides a wide range of tools and libraries to manipulate the policy, build the policy, debug policy violations, and more.
The SELinux support in OpenWrt comes in two parts: a number of additional packages for various libraries and applications, and some integration work in OpenWrt. We will cover both in the next sections. It is worth mentioning that our work does not provide a SELinux policy specifically modified or adjusted for OpenWrt: we simply use the SELinux reference policy, which users will have to tune to their needs.
SELinux-related packages
Getting SELinux to work required a number of new packages to be added in OpenWrt. Those packages were contributed to the community-maintained package feed at https://github.com/openwrt/packages/. They were initially submitted through the mailing list, and then submitted as a pull request.
selinux-python, a number of SELinux tools written in Python, especially audit2allow for policy debugging.
SELinux integration
Our second patch series, for OpenWrt itself, allows to build a SELinux-enabled system thanks to the following changes:
Allow to build Busybox with SELinux support, so that all the Busybox applets that support SELinux specific options such as -Z can be built with libselinux (patch 1/7)
Add support in OpenWrt’s init application, called procd, for loading the SELinux policy at boot time (patch 2/7). This patch has been submitted separately for integration into the procd project.
Add support for building a new host tool called fakeroot (patch 3/7).
Add support for building squashfs images with extended attributes generated by SELinux setfiles tool (patch 4/7). This is why fakeroot is needed: writing those extended attributes that store SELinux security contexts require root access, so we run the entire process within a fakeroot environment. This also requires building the squashfs tools with extended attributes support (patch 7/7).
Add new options to enable in the Linux kernel support for SELinux and SquashFS with extended attributes (patch 5/7 and patch 6/7).
Conclusion
Integrating those two security features in OpenWrt required numerous changes in the build system, and the corresponding patches are still under review by the OpenWrt community. We hope to see these features merged in 2020.
The Embedded Linux Conference Europe edition 2019 took place a few weeks ago in Lyon, France, and no less than 7 engineers from Bootlin attended the conference. We would like to highlight a selection of talks that Bootlin engineers found interesting. We asked each of the 7 engineers who attended the event to pick one talk they liked, and make a small write-up about it. Of course, many other talks were interesting and what makes a talk interesting is very subjective!
Introduction to HyperBus Memory Devices, by Vignesh Raghavendra
Talk selected by Gregory Clement
Vignesh started his talk by presenting the HyperBus from the hardware point of view. It is a bus using 8 data lines, using either a single or a differential clock as well as a bi-directional data strobe. These last two features clearly indicate that the designers of this bus seek high bandwidth. Two types of memory are available: HyperRAM and HyperFlash and the talk focused on the second one.
The read throughput of the HyperFlash can reach 400MB/s, it is compatible with SPI flash and is an alternative to the octal SPI NOR flashes. Then Vignesh presented the transactions done on the bus for the flash, which is very similar to what is done on SPI bus. He also compares the traditional parallel CFI flashes to the HyperFlash. And finally he describes the 2 types of controllers: Dedicated HyperBus Controllers and Multi IO Serial controllers.
In the last part of the talk, Vignesh presented the recently add kernel features, and future improvements.
This talk was a good introduction to this new bus, covering the hardware parts as well as software support in the Linux kernel.
Open Source Graphics 101: Getting Started, by Boris Brezillon
Talk selected by Paul Kocialkowski
During this talk, Boris provides a comprehensive and accessible overview of the graphics stack that supports GPUs in systems based on the Linux kernel. He also provides insight about the inner workings and architecture of GPUs. Although Boris defines himself as a not (yet) experienced Graphics developer, his talk contains all the elements needed to get a clear first idea on the topic as he covers hardware, kernel and userspace aspects.
To begin with, he explains how the GPU pipeline is split into multiple stages that are needed one after the other to generate a final image from a set of 3D models. The first stage is geometry and involves operations on the vertices that compose the 3D models, followed by rasterization where the view of the 3D scene is materialized on a 2D viewport, producing the end image. He also presents the concept of shaders: they are small dedicated programs that run on the GPU to make each stage configurable and flexible in order to produce the exact wanted result.
He then provides details about how GPU architectures implement massive parallelization to provide efficient results and also details some of the pitfalls that can occur with this approach. After that, Boris presents how the main CPU interacts with the GPU, introducing the concept of a command stream to submit jobs to the GPU.
With all these concepts laid out comes the time for him to present how the software stack is organized to support GPUs. After a general overview, specific aspects are presented. This includes graphics APIs such as OpenGL and Vulkan (and how they follow distinct paradigms) but also covers topics related to Mesa internals such as intermediate representations or windowing system integration. Kernel aspects are not forgotten either and a rationale regarding the (unusual) kernel/userspace separation in place for GPUs is also provided to clarify prominent design choices.
This talk succeeds at providing an introduction to GPUs and 3D rendering that can be understood without specific graphics knowledge while also giving a good idea of how the supporting software stack is organized. It is highly recommended for anyone interested in learning more on these topics!
Learning the Linux Kernel Configuration Space: Results and Challenges, Matthieu Acher
Talk selected by Michael Opdenacker
TuxML (Linux and Machine Learning) is an open-source research project aiming at exploring the Linux kernel configuration space through machine learning. With more than 15,000 configuration parameters, the Linux kernel now has up to 106000 possible configurations. Compare this figure to 1080, the approximate number of atoms in the universe.
As it is not possible to test all such configurations (all the more as each takes about 10 minutes to compile), the goal of the project is to predict “interesting” configurations, that could expose distinct bugs.
Starting from random configurations (from make randconfig), they use statistical learning to eventually pinpoint sets of parameters causing build failures, and avoid testing configurations that are expected to fail. This way, TUXML can be more efficient in exploring the configuration space and find bugs.
That’s typically where researchers can help us engineers and Linux kernel contributors. You need a solid theoretical background in machine learning to process data efficiently.
On Linux 4.13, the research team has managed to explore more than 15,000 different configurations through more than 95,000 hours of computing, eventually to find (and fix) 16 bugs in Linux. Some of these bugs may not come as a surprise for experienced kernel developers, but some others could expose unexpected issues that a human user may not find spontaneously.
They are also trying to use their data to predict the impact of configuration parameters on kernel size, but it turns out that size is hard to predict. At least, they managed to find “influential” options, some expected ones, and some less expected ones, deserving further investigation.
This project looks definitely useful for improving the test coverage of the Linux kernel, by working smarter than trying to compile purely random configurations. Your help is needed for testing, investigating and fixing kernel bugs, and giving your feedback.
Sergio gave a talk about debugging with an interesting approach: he started by acknowledging that today, printk is very widely used to do serious debugging but that in some cases it would be much more efficient to use other tools. Indeed there are plenty of open source tools available out there so why don’t we use them?
He presented a table indicating, from his point of view, which tool would best fit a given problem and then enumerated a few tips and commands that everybody can use to understand what went wrong in their kernel, for instance after a panic.
Is addr2line the best way to avoid printk messages right after a panic? or maybe the Linux script faddr2line? or even GDB? Maybe you don’t have access to the panic trace yet, in this case you could be interested in looking at pstore or kdump?
Or maybe an issue will more efficiently be hunted with tracing, in this case Sergio shown several static and dynamic options: using tracepoints, kprobes, ftrace, and proposed many others.
Lock-ups and memory leaks are also covered in the slides (see below), but not in the video because unfortunately the 35 minutes slot allowed was not enough for Sergio to detail all these interesting debugging methods. We wish he had more time to give all his feedback around these underused -while powerful- tools!
Supporting Video (de)serializers in Linux: Challenges and Works in Progress
Talk selected by Thomas Petazzoni
Luca Ceresoli’s talk at this ELCE is a good example of an interesting talk, as it combines an introduction to new hardware, what is the status of the support for this hardware in Linux, and what are the challenges to overcome to complete the integration of the hardware support in the kernel, with some open discussion.
Luca’s talk was about the support for video serializers and deserializers, with a focus on camera support. Cameras are usually connected to a system-on-chip using a parallel interface, a MIPI CSI interface or some LVDS interface. However, these interfaces only work for very short distances between the system-on-chip and the camera, and may not work well in electromagnetically noisy environments. For such situations, there are some technical solutions that consists in serializing the camera stream in a fast robust link (typically a coax cable) and then deserialize it before it is captured by the SoC through a standard camera interface. This fast robust link of course transports the stream data itself, but can also transport control information (GPIO status, I2C bus to talk to the remote camera sensor).
There are two main technologies today implementing this: the GMSL technology from Maxim and the FDP-LinkIII technology from Texas Instruments. Luca’s focus is on the latter technology, since that’s what he has been working on for the past months.
After this hardware introduction, Luca gave a status of the different patch series that have been posted by various contributors (himself included) on the Linux kernel mailing lists: some preliminary support for GMSL has been posted by Kieran Bingham, and some preliminary support for FDP-LinkIII has been posted by Luca.
Luca then presented the ideal implementation to support these interfaces, but then quickly dived into the troubles and tribulations: there is no support for stream multiplexing in Video4Linux currently, there is no support for parts of a V4L pipeline going faulty, and there is no support for hotplugging in V4L.
Then, there are some challenges with how to handle the remote I2C bus offered by those serializers/deserializers. Since camera sensors often have the same I2C address, the serializers/deserializers often have some sort of “solution” to this: an I2C switch in the GMSL (de)serializers, and a translation table for I2C addresses in FDP-LinkIII (de)serializers. Luca discussed how these are currently supported in Linux.
At the end of the talk, quite a bit of discussion took place, both about the V4L issues and the I2C issues raised in Luca’s talk. Overall, it was a useful talk if you’re interested in this specific topic.
As someone not very familiar with the V4L2 Framework, I was pleasantly surprised that Hans’ talk was done in such a way that both experienced and beginner developers were able to follow.
Hans started with a quick introduction to the various concepts of video encoding and decoding that needs to be understood to follow the highly technical explanations of the current status of stateless and stateful codecs support.
Besides describing the technical challenge of implementing such support, Hans gave a good overview of the challenges that are faced by the community, focusing on the necessity of having good testing tools, such as the new vicodec driver.
He described the complexity of implementing support for Stateless decoders (where the hardware decoder doesn’t keep track of the state, this has to be done in software), and explained that the new Request API is a good step towards achieving such support, with 2 decoders already supported in the staging area.
Hans then explained the userspace APIs that are to be used when dealing with Stateless decoders, starting some interesting discussions along the way.
All in all, such a talk is a good example of how we can use events such as ELCE to both give good technical insight on existing frameworks, but also to trigger discussions about the ongoing and future work amongst the active developers and maintainers that are brought together by the event.
One Build to Rule Them All: Building FreeRTOS & Linux Using Yocto
Talk selected by Alexandre Belloni
In this talk, Alejandro Hernandez explains how to build FreeRTOS using OpenEmbedded.
He starts by explaining the use case for building both FreeRTOS and a Linux system using the same build system, in this case OpenEmbedded.
He then shows the meta-freertos layer he developped to get OpenEmbedded to build FreeRTOS. The toolchain he used is fairly classic with GCC, binutils, gdb. The main difference is that newlib is used as the C library.
meta-freertos then depends on previous work that has been done, integrating a newlib and libgloss recipes in oe-core. The core of meta-freertos is then a class, freertos-app.bbclass, allowing to abstract many details allowing to build a FreeRTOS application and image for the target. A poky-freertos distribution configuration is also provided.
Alejandro then demoes multiple FreeRTOS applications.
Finally, he goes over multiconfig, the multiple configuration build dependencies, allowing OpenEmbedded to build an image using a configuration but depending on tasks using a different configuration. In other words, this allows to build a Linux system image after building a FreeRTOS application so it can be included in the image. This is very useful in the case Linux is running on the application processor and needs to load FreeRTOS on a smaller processor.
Authenticate and Encrypted Storage on Embedded Linux
Talk selected by Kamel Bouhara
Jan’s talk is introducing us to the current authentication and encryption methods that go on top of the Linux storage stack.
He started reminding us some basic crypto terminologies and then depicted all the existing technologies by the storage they fit into.
For block device storage dm-verity is a good choice to verify integrity of read-only filesystems and the verification is done on each node of a hash tree. For a file or application based verification fsverity is a more relevant tool as it allows on-demand verification.
On raw NAND devices, the integrity should be checked using the UBI filesystem associated to an HMAC or image signature authentication with a root key. This solution can be completed with fscrypt to encrypt specific data on the filesytem.
For the encryption stage, Jan mentioned the ecryptfs project, which is not maintained anymore and could be well replaced by fscrypt which allows to hold several keys in the same filesystem in a multi-user environment. It is therefore a good alternative to dm-crypt which is a block-based based encryption solution used on large block devices and it is not protected against replay attacks using old blocks.
For a TPM based authentication, he recommendeds using the kernel integrity subsystem called IMA/EVM, which is a layout on top of other filesystem, the project Keylime is good example for this: https://sched.co/TLCY.
Jan shared some good practices on how to manage the Master key storage like not using a password based key, if possible use hardware capabilities like ARM TrustZone and OPTEE and use of a verified boot and key wrapping for the master key.
With 8 engineers participating to the Embedded Linux Conference Europe, almost the entire Bootlin engineering team took part to the conference. As usual, we not only attended the event, but also contributed by giving a total of 5 talks and 2 tutorials, for which we’re happy to share below the videos and slides. Also, as part of this conference, Bootlin CTO Thomas Petazzoni received an award for his contribution to the conference.
Tutorial given by Michael Opdenacker, slides in PDF, slides source document. The video is not yet available, but should be published in the future.
Introduction to the Buildroot embedded Linux build system
Tutorial given by Thomas Petazzoni, slides in PDF, slides source code. The video is not yet available, but should be published in the future.
Award to Thomas Petazzoni
During the traditional closing game of the conference, we were really happy to have Bootlin’s CTO Thomas Petazzoni called on stage, to receive from the hands of Tim Bird, an award for his continuous 11 year participation to the conference, with 24 presentations given, one keynote and for the past two years, participation to the conference program committee. We are honored and proud by this recognition of Thomas contribution to the conference.
Bootlin will be preent at Embedded World 2020, in Nuremberg on February 25-27. We will be present on STMicroelectronics booth in hall 4A, stand 138. We will have two demos of the STM32MP1 platform running Linux, and of course details about Bootlin services around embedded Linux and Linux kernel development and training.
This time around, we’re quite late to the party, but Linux 5.4 was indeed released a number of weeks ago, and once again, Bootlin contributed a number of patches to this Linux kernel release. As usual, the most useful source of information to learn about the major features brought by Linux 5.4 are the LWN articles (
