2021 has come to an end, a year that everyone will most likely consider as somewhat complicated and unusual, even though the current situation seems to now becoming the new normal. The switch to a new year is generally a good moment to take a step back, and review what happened in the past year, and draw some directions for the coming year.
In this blog post, we’d like to do exactly this for Bootlin, which has seen a number of significant changes this year, as well as a continuation of its usual activities.
We also take this opportunity to wish you all an happy year 2022, and send you our best wishes. May 2022 be full of interesting projects and also be a safe year for everyone.
Here are the main topics that we cover in this lengthy blog post:
Acquisition and recruiting
One major change for Bootlin in 2021 is that the company was acquired by Thomas Petazzoni, former CTO, and Alexandre Belloni. So it’s an internal acquisition, by former employees, meaning that Bootlin has kept the same offering and spirit. Further, Bootlin’s original founder Michael Opdenacker is still in the team, but now as an employee. See our blog post regarding the acquisition, back in February.
In 2021, we also recruited several engineers, bringing significant additional expertise to our team:
- Thomas Perrot, who joined just before the start of 2021, bringing 10 years of experience in embedded Linux BSP development, with a strong Yocto expertise.
- Hervé Codina, who joined in March 2021, bringing 20+ years of experience in bare-metal and Linux embedded development.
- Clément Léger who joined in June 2021, bringing 10 years of experience in Linux kernel development, including the expertise on porting the Linux kernel to a brand new CPU architecture.
We continue to have open positions for embedded Linux engineers, and we plan to hire 3 to 4 engineers in 2022, hiring for the first time engineers located outside of France.
Engineering projects
As is the case every year, our engineering team has been kept busy this year mostly by our engineering projects, all focused on our core expertise of low-level embedded Linux development.
We have selected below a few highlights of our work in 2021, in various areas.
Build systems
Apart from delivering numerous BSPs based on Yocto or Buildroot to our customers, we have also directly contributed to both the Yocto and Buildroot open-source projects.
On the Yocto Project side:
- Bootlin engineer Michael Opdenacker has become one of the co-maintainers of the official Yocto Project documentation, making numerous improvements and contributions to this documentation. See our blog post on this topic. We will continue this involvement in the Yocto Project documentation in 2022.
- Bootlin engineer and COO Alexandre Belloni has been active in the build and release engineering effort of the Yocto Project, as a member of the Yocto Project SWAT team. Alexandre has been working directly with the main Yocto Project architect, Richard Purdie, on the review and validation of new contributions. Other Bootlin engineers have also helped in investigating and resolving specific bugs. We will also continue this involvement in the Yocto Project build engineering in 2022.
On the Buildroot side:
- We have continued our work towards the implementation of top-level parallel build support in Buildroot. Bootlin engineer Hervé Codina has posted several iterations of a patch series bringing a mechanism preventing file overwrites between packages, a requirement for proper top-level parallel build.
- We implemented and published, in partnership with ST, a Buildroot
BR2_EXTERNALcalled buildroot-external-st which contains example configurations for the STM32MP1 platforms from ST. - We contributed support for hybrid ISO9660 images, supporting in a single image PC platforms based on legacy BIOS, 32-bit UEFI BIOS and 64-bit UEFI BIOS, using the grub2 bootloader.
- We helped one of our customers reduce their out-of-tree Buildroot patches by upstreaming a number of new Buildroot packages.
Outside of our the Yocto Project and Buildroot, we also ventured into the world of Debian/Ubuntu for embedded systems, by using the ELBE tool, to which we contributed support for building Ubuntu-based images. See our blog post that describes how to use ELBE to automatically build Ubuntu-based images for the Raspberry Pi.
Audio support in Linux
Audio support in the Linux kernel is another area where Bootlin engineers have specific expertise with. Here are some examples of audio related projects we worked on in 2021:
- Integration of audio support for an i.MX6 platform with a complex dual AD1978 audio codec configuration, with a TDM8 audio interface between the SoC and the audio codecs.
- Development of a brand new and complete ALSA driver for a new PCIe sound card, based on a FPGA.
- Integration of audio support for an i.MX8 platform used in the automotive space with a complex audio configuration that involves multiple A2B transceivers through the Analog Devices AD2428 A2B master, Bluetooth audio, and more.
OP-TEE
In 2021, we started working on a major project: adding support for the Microchip SAMA5D2 processor to the OP-TEE project. This project is already well underway, as we have a functional port of OP-TEE, which is now being upstreamed.
As part of this, Bootlin engineer Clément Léger has implemented and contributed a generic clock framework for OP-TEE, which has already been accepted upstream. See our blog post for more details.
Clément also gave a talk at the Embedded Linux Conference 2021 on the topic of OP-TEE, titled OP-TEE: When Linux Loses Control. Slides and video are available.
We expect to continue the upstreaming of the SAMA5D2 support in 2022, and also develop support for additional SAM5D2 hardware capabilities in OP-TEE.
U-Boot extension board manager
In collaboration with the BeagleBoard.org community, we developed and contributed to U-Boot a generic extension board manager. This mechanism allows hardware platforms such as the BeagleBone to automatically detect extension boards that are connected, and apply the Device Tree overlays that provide the hardware description for those extension boards.
This work was covered in detail in the talk given by Bootlin engineer Köry Maincent at the Live Embedded Event in June 2021, see the slides and video.
Modems
In 2021, we had the opportunity to work on several projects that involved 3G/4G/5G modems, strengthening our knowledge of the modem stack in Linux, especially around modem-manager, libmbim and libqmi.
For example, we worked on the support of a Quectel BG95 modem interfaced with a RaspberryPi 4, or the support of a Sierra Wireless EM9190 modem interfaced over PCIe to an NXP i.MX6. The latter was particularly challenging and is still on-going, as the upstream Linux support for PCIe 5G modems is still very recent. We contributed a few fixes in this area.
Secure boot
Secure boot remains an important topic for a growing number of projects, and in 2021, we helped customers with secure boot on several i.MX6 platforms, and on one i.MX8 platform. The Yocto integration of secure boot was also a key aspects in those projects, to get a proper process for signing and verifying all states of the boot process. We also have a few U-Boot contributions in our contribution pipeline related to improving secure boot support. See for example the talk from Thomas Perrot at Live Embedded Event, relating the secure boot setup on i.MX8.
Camera, ISP and video support
We finished 2020 with a lot of on-going contributions to the Linux Video4Linux subsystem, which we had summarized in a blog post early 2021. In particular, our drivers for the OV5568 and OV8865 camera sensors were merged upstream.
In 2021, Bootlin engineer and multimedia expert Paul Kocialkowski continued to work on several multimedia topics. Some of the key projects included:
- Optimizing the H264 decode → rescale → H264 encode pipeline of one of our customers, on an Allwinner H3 platform. We were able to significantly optimize the pipeline by leveraging the H264 decoder built into the Allwinner processor, for which Paul had written a Linux kernel driver several years ago, and by developing a custom ffmpeg plugin that offloaded the rescaling to the GPU, using the open-source lima support. A fully open-source solution!
- Developing a proper Linux kernel driver for the Allwinner ISP, with support for debayering and noise filtering. This was a significant challenge as the Allwinner ISP was so far only supported through closed-source binary blobs. We have already submitted a first iteration to the upstream Linux kernel community. See our blog post for more details, as well as the talk given by Paul at the Embedded Linux Conference: slides and video.
- Improving the Allwinner Linux kernel camera driver to support single buffer capture. This is useful in scenarios where camera sensors are not used to capture a sequence of frames, but just single frames, and the available memory is limited. This is particularly true on platforms such as the Allwinner V3s, which may be limited to just 64MB of RAM. This is going to be submitted upstream soon, as part of our on-going work on the Allwinner camera driver.
NAND and flash support
With Bootlin engineer Miquèl Raynal being a maintainer of the NAND subsystem in Linux and a co-maintainer of the MTD subsystem, it should be no surprise that we have continued to deal with a number of flash memory related projects in 2021:
- We have brought in mainline the support for several NAND flash controllers:
- For the Arasan NAND controller, used in some Xilinx processors. See our blog post on this topic.
- For the ARM Primecell PL35x NAND controller, used in some other Xilinx processors. See our blog post on this topic.
- For the NAND controller found at least in the the Renesas R-Car Gen3 and RZ/N1 processors.
- Contributed support for the NV-DDR interface, which is used by some NAND flash chips and controllers to increase the throughput. See our blog post.
- Continued our work on generalizing ECC support in the MTD subsystem, so that all ECC setups (software ECC, on-die ECC, ECC in the NAND controller, or external ECC engine) can be supported regardless of the NAND interface (serial or parallel). See the talk from Miquèl Raynal, ECC engines given at last year’s ELCE.
- Improved the TI GPMC NAND controller driver to properly support NAND chip with large pages (larger than 4KB).
ADC/IIO support
We’ve helped a few customers with Linux kernel support for ADC devices, namely:
- Extend the support for the MAX1027 family of ADCs, with support for external triggers
- Bring support for the ADC found in TI AM437x processors
All of these were contributed to the upstream Linux kernel, Miquèl Raynal also wrote a extensive blog post on various aspects of the IIO subsystem and has taken the opportunity of these projects to also improve/clarify various aspects of the IIO core in a recent patch series.
Networking
Networking support in the Linux kernel is also one area where Bootlin is very active. Here are some key projects we worked on in 2021, some of them being on-going with additional work expected in 2022:
- We implemented support for PTP offloading for the Qualcomm AR803X PHY.
- We started working on QUSGMII support in Linux, a new standard that allows PTP time-stamps to be included directly on the preamble of QSGMII frames.
- We started working on the 802.15.4 stack in the Linux kernel, with our initial step being support for passive scanning.
- We implemented mqprio support in the mvneta Linux kernel driver, used for some Marvell platforms. mqprio is a queuing discipline that allows mapping traffic flows to hardware queue ranges using priorities and a configurable priority to traffic class mapping.
- We implemented Frame DMA support in the Ocelot Ethernet switch driver, significantly improving the performance of frame injection/extraction by the CPU into/from the switch traffic.
- We developed proper SFP support in Linux and U-Boot, with dynamic reconfiguration (in Linux only), for a customer using the Zynq 7000 platform and a complex network setup.
BSP development
Many of our projects are obviously related to the development of update of complete Linux BSP for our customers (bootloader, Linux kernel, custom embedded Linux distribution). Here are a few examples:
- Development of a brand new U-Boot, Linux and Yocto based BSP to migrate an existing product running Windows CE on a ST Spear320 processor to Linux. This sort of development proved to be challenging as the Spear320 support in both U-Boot and Linux is close to be abandoned (and in fact has since then been removed from U-Boot upstream). We hope to be able to contribute to improve the upstream Spear320 support in 2022.
- Migration of an AM335x/Buildroot based BSP to newer Linux kernel and Buildroot releases, for a customer in the healthcare industry
- On-going migration of an i.MX6 BSP from an old kernel release and a custom build-system to recent versions of U-Boot and Linux as well as a proper standardized Yocto Project based Linux distribution. This project presents some interesting challenges as it uses only one of the two Cortex-A9 cores to run Linux, the other one runs a bare-metal application, and we will also migrate this to a proper usage of the remoteproc and rpmsg mechanisms.
- We worked with several customers on STM32MP1 platforms, helping with porting on new platforms, extending the Device Tree and device drivers, developing custom Yocto-based or Buildroot-based distributions. STM32MP1 is definitely becoming a popular platform for a number of new projects.
- We also worked with several customers on custom platforms based on the RasberryPi Compute Module, doing Device Tree configuration/tweaking and Yocto integration.
Over-the-Air update integration
We continued to help our customers with the integration of Over-The-Air update solutions in their embedded Linux systems. A few examples:
- We integrated RAUC for an i.MX6 platform, using the Barebox bootloader and Yocto Project based distribution. Bootlin engineer Kamel Bouhara wrote a detailed blog post on this topic, as well as the talk from Kamel at Live Embedded Event
- We integrated the Azure Device Update for IoT Hub, using the U-Boot bootloader on a RaspberryPi CM4 platform, also with a Yocto Project based distribution. Internally, the Device Update for IoT Hub is based on swupdate
- As part of a migration of an existing embedded product based on Debian to a Yocto-based distribution, we are integrating Mender.
Crédit Impôt Recherche
For our French customers, another important milestone achieved by Bootlin in 2021 is the delivery of our Crédit Impôt Recherche agreement, which allows our French customers to benefit from tax incentives on research and development activities done by Bootlin for their projects. See our blog post for more details.
Training
With the COVID19 still making travel conditions difficult and uncertain, most of our training activity in 2021 was dedicated to on-line training courses. Indeed, 69 out of our 72 training sessions were delivered online this year. In total, we delivered our training courses to 906 engineers in 2021.
In 2021, we published two new training courses:
- Our Embedded Linux boot time optimization training, which teaches engineers how to measure and optimize the boot time of the different components of an embedded Linux system.
- Our Real-time Linux with PREEMPT_RT training, which teaches engineers on how to build, run and debug real-time Linux systems using PREEMPT_RT
Just like all our other training courses, the complete training materials for those new courses are freely available, distributed under the CC-BY-SA license.
Another major event in 2021 was our work to get the French Qualiopi certification, which proves the quality of our training organization and processes. As part of this, we have improved several aspects of our training courses, mainly regarding feedback collection and handling as well as the evaluation of the participants.
In 2022, we expect to at least:
- Improve how our online course are delivered, by ensuring more participants can do the practical labs by themselves, which is an important part of the learning process
- Publish at least one more training course. We already have plans for a course on a topic that we think will be very relevant to many embedded Linux engineers. Stay tuned for updates on this!
- Have additional capacity to deliver our training courses, which are seeing significant demand.
For more details about our training offering, see our training page, which details our dedicated on-line and on-site sessions as well as our public on-line sessions.
Contributions
In Bootlin continued and strong open-source focus, we once again contributed to different open-source projects in 2021:
- Linux kernel. The best summaries are our blog posts about our contributions to Linux 5.10, Linux 5.11, Linux 5.12, Linux 5.13, as well as Linux 5.14 and 5.15. Some of our significant contributions: huge effort on ECC engine support in the MTD subsystem, rv3032 RTC driver, support for new MIPS platforms from Microchip, a Simple Audio Mux driver, major work on timer/TCB support on Microchip ARM platforms, new I3C master controller driver for the Silvaco I3C IP, new drivers for the OV5568 and OV8865 camera sensors, NAND controller drivers for the Arasan, PL35x and Renesas IPs, enabling of ADC support on TI AM437x.
- U-Boot. Our biggest contribution has been the extension board manager, which is described earlier in this blog post. We also contributed various small fixes and improvements.
- OP-TEE. We contributed a generic clock framework, and the support for the SAMA5D2 clock tree and TRNG driver. So far 55 commits from Bootlin have been integrated in OP-TEE.
- Buildroot. We contributed a total of 257 patches. The main contributions are in the area of tooling to list security vulnerabilities in Buildroot packages, support for Bootlin external toolchains, improved SELinux support, support for the Beagle-V RISC-V platform, a good number of new packages, support for hybrid ISO9660 images. In addition, Thomas Petazzoni continued his work as a co-maintainer of the project: Thomas has reviewed and merged close to 2000 patches from Buildroot contributors throughout 2021.
- Yocto Project. As outlined above in this blog, we did major contributions to the Yocto Project, with Michael Opdenacker becoming co-maintainer of the Yocto Project documentation, and Alexandre Belloni being involved in the build and release engineering effort.
- Linux Test Project. We fixed a number of issues in LTP tests that prevented from using LTP in embedded-oriented systems, such as the ones that can be generated using the Yocto Project. See our blog post for more details.
- Our training materials, which are all freely available, have seen no less than 872 commits. These includes updates to our existing training courses, but also our new Real-Time Linux with PREEMPT_RT training course.
- Our freely available toolchains have also been improved and updated. See our blog post.
- We continued to maintain our very popular Elixir code indexing tool.
Conferences
Here are the talks that we presented this year, at various virtual events:
- FOSDEM 2021
- Live Embedded Event
- Security vulnerability tracking tools in Buildroot, by Thomas Petazzoni. Slides and video.
- Secure boot in embedded Linux systems, by Thomas Perrot. Slides and video.
- Understanding U-Boot Falcon Mode, by Michael Opdenacker. Slides and video.
- Device Tree overlays and U-Boot extension board management, by Köry Maincent. Slides and video.
- Getting started with RAUC, by Kamel Bouhara. Slides and video.
- Embedded Linux Conference
- I3C in tomorrow’s design, by Miquèl Raynal. Slides and video.
- Embedded Linux nuggets found in Buildroot package Eldorado, by Michael Opdenacker. Slides and video.
- OP-TEE: When Linux Loses Control, by Clément Léger. Slides and video.
- Advanced Camera Support on Allwinner SoCs with Mainline Linux, by Paul Kocialkowski. Slides and video.
- A webinar organized in partnership with ST, Device Tree 101, by Thomas Petazzoni. Slides and video.
- A webinar organized in partnership with Microchip, Improving Linux Boot Time in Your Embedded Application, with the participation of Thomas Petazzoni. Link.
