Luca Ceresoli joins Bootlin team

Welcome on board!The entire team at Bootlin is extremely happy to welcome Luca Ceresoli, who started working with us on March 1, 2022. Based in Italy, Luca is the first employee of Bootlin based outside of France, and we plan to continue to expand our hiring in a similar way in the future.

Luca brings a vast amount of embedded and embedded Linux expertise to the Bootlin team. Luca has been working on embedded products since 2002, and embedded Linux products since 2008. He has helped design, develop and ship products based on a large number of embedded processors, developing complete BSPs, developing Linux kernel drivers, integrating complete systems using Buildroot or Yocto. Luca has also contributed to the Linux kernel, Buildroot, and several open-source projects, and has spoken several times at FOSDEM, the Embedded Linux Conference, and other events.

Luca’s arrival at Bootlin not only means that we have more capacity to help our customers with their embedded Linux projects, but also that we have more capacity to deliver training courses, and a new ability to deliver some of our training courses in Italian.

Once again, welcome on board Luca!

Bootlin contributions to OP-TEE 3.15 and 3.16

OP-TEE logoLast year, Bootlin started contributing to the OP-TEE project, which is an open source Trusted Execution Environment (TEE) implemented using the Arm TrustZone technology. We published a blog post about our contribution of a generic clock framework to OP-TEE, and also presented a talk OP-TEE: When Linux Loses Control (slides, video).

As part of this work, Bootlin engineer Clément Léger contributed to the OP-TEE project, and many of his contributions have already been merged, and released as part of the 3.15 and 3.16 OP-TEE releases. In this blog post, we present some details of our contributions to OP-TEE so far, and our next steps.

Summary

Since then, we contributed a number of features and improvements to OP-TEE. First, a number of generic, HW-agnostic contributions:

The Microchip SAMA5D2 platform support was also greatly improved with the following improvements:

  • Cleanup of existing SAMA5D2 support
  • Rework of the memory layout
  • Device Tree support
  • Board support for sama5d27-som1-ek1
  • Clock tree support: includes drivers for all the clocks available on sama5d2 SoC
  • TRNG driver
  • Reset and shutdown controller drivers
  • PSCI support for reset and shutdown

Contribution details

We contributed 11 commits to OP-TEE 3.15.0:

We contributed 48 commits to OP-TEE 3.16.0. This level of contribution makes Bootlin engineer Clément Léger the second most active contributor by number of commits for this OP-TEE release.

Next steps

We will continue our effort on sam5d2 support on OP-TEE and as part of this, there will be contributions on several generic subsystems as well as SAMA5D2 support:

  • Watchdog support
    • Generic watchdog API
    • OP-TEE Watchdog service compatible with arm,smc-wdt Linux driver
    • Sama5d2 watchdog driver
  • RTC support
    • Generic RTC API
    • OP-TEE RTC PTA to expose RTC to Linux
    • sama5d2 RTC driver
    • Linux driver for OP-TEE RTC
  • SAMA5D2 suspend support
    • Support forULP0, ULP1, ULP0 Fast and backup modes
    • PSCI support
  • SAMA5D2 interrupt controller support

Do not hesitate to contact us if you need help and support to integrate or deploy OP-TEE on your platform, either Microchip platforms, but also other ARM32 or ARM64 platforms.

Linux kernel drivers and Yocto public online courses now available for US/America time zones

Since the start of the pandemic in March 2020, we have been offering our popular training courses online to our customers, in both public sessions (opened to individual registration) and dedicated sessions (organized on-demand for our customers, at their choice of date/time).

Our public sessions were initially all organized in the afternoon of Europe (14:00 to 18:00 Paris time), but we started in late 2021 proposing our Embedded
Linux system development training course
for customers in the US/America, especially on the West Coast. Thanks to the success of this, we have decided to also start offering at times convenient for US/America customers the following courses:

The chosen time is ideal for US/America customers, but also works for participants in Europe who would like to attend our courses outside of their normal working hours.

Of course, we have plenty of other training sessions scheduled, for all our 7 different training courses, covering a wide range of time zones. Check our training page for all the details, and contact us if you have specific needs: a dedicated training course for your company, a session organized at a different date/time, etc.

Linux 5.16 released: Bootlin contributions

Linux 5.16 has been released on January 9. As usual, our recommended reading to learn more about this release is the corresponding Kernelnewbies.org page and the two articles from LWN covering the 5.16 merge window: part 1 and part 2.

As usual, Bootlin contributed a number of patches to this release, with a total of 117 commits, making Bootlin the 22th contributing company according to statistics (Unknown and Hobbyists not counting as companies).

Here are the main highlights of our contributions to 5.16:

  • Alexandre Belloni, as the maintainer of the RTC subsystem, continued to improve the core subsystem and RTC drivers. He added a new user-space interface to RTC to allow getting/settings RTC parameters. It is used to get the features of an RTC from user-space, to get/set the RTC correction or to configure Backup Switch Mode. In addition, Alexandre made various improvements to several RTC drivers, such as adding Backup Switch Mode, and general refactoring.
  • Clément Léger did a small fix in the clock drivers for Microchip ARM platforms, fixing an issue discovered as part of his work porting OP-TEE to Microchip ARM platforms.
  • Hervé Codina made some fixes to the fsmc NAND controller driver, which is used on a number of old platforms from ST. They fix support of certain NAND chips on those platforms. These issues were discovered as part of the development of a Linux BSP for an old ST Spear320 platform.
  • Maxime Chevallier fixed a deadlock in the network stack, that was causing the kernel to stop booting when using a root filesystem over NFS combined with the network interface using a SFP module.
  • Miquèl Raynal contributed many improvements to the max1027 ADC driver in the IIO subsystem, supporting hardware triggers.
  • Miquèl Raynal contributed support for the ADC found on Texas Instruments AM437x platforms. This required significant changes in the MFD driver that is used to support the multiple features of this ADC, as well as improvements in the existing IIO driver for TI ADCs.
  • Paul Kocialkowski contributed a small addition enabling the Rockchip VPU on Rockchip PX30 platforms, and merged the Device Tree bindings for the logiCVC display controller (but not yet the driver itself).

And now, the complete list of commits:

2021 at Bootlin, a year in review

2021 has come to an end, a year that everyone will most likely consider as somewhat complicated and unusual, even though the current situation seems to now becoming the new normal. The switch to a new year is generally a good moment to take a step back, and review what happened in the past year, and draw some directions for the coming year.

In this blog post, we’d like to do exactly this for Bootlin, which has seen a number of significant changes this year, as well as a continuation of its usual activities.

We also take this opportunity to wish you all an happy year 2022, and send you our best wishes. May 2022 be full of interesting projects and also be a safe year for everyone.

Here are the main topics that we cover in this lengthy blog post:

Acquisition and recruiting

One major change for Bootlin in 2021 is that the company was acquired by Thomas Petazzoni, former CTO, and Alexandre Belloni. So it’s an internal acquisition, by former employees, meaning that Bootlin has kept the same offering and spirit. Further, Bootlin’s original founder Michael Opdenacker is still in the team, but now as an employee. See our blog post regarding the acquisition, back in February.

In 2021, we also recruited several engineers, bringing significant additional expertise to our team:

  • Thomas Perrot, who joined just before the start of 2021, bringing 10 years of experience in embedded Linux BSP development, with a strong Yocto expertise.
  • Hervé Codina, who joined in March 2021, bringing 20+ years of experience in bare-metal and Linux embedded development.
  • Clément Léger who joined in June 2021, bringing 10 years of experience in Linux kernel development, including the expertise on porting the Linux kernel to a brand new CPU architecture.

We continue to have open positions for embedded Linux engineers, and we plan to hire 3 to 4 engineers in 2022, hiring for the first time engineers located outside of France.

Engineering projects

As is the case every year, our engineering team has been kept busy this year mostly by our engineering projects, all focused on our core expertise of low-level embedded Linux development.

We have selected below a few highlights of our work in 2021, in various areas.

Build systems

Apart from delivering numerous BSPs based on Yocto or Buildroot to our customers, we have also directly contributed to both the Yocto and Buildroot open-source projects.

On the Yocto Project side:

  • Bootlin engineer Michael Opdenacker has become one of the co-maintainers of the official Yocto Project documentation, making numerous improvements and contributions to this documentation. See our blog post on this topic. We will continue this involvement in the Yocto Project documentation in 2022.
  • Bootlin engineer and COO Alexandre Belloni has been active in the build and release engineering effort of the Yocto Project, as a member of the Yocto Project SWAT team. Alexandre has been working directly with the main Yocto Project architect, Richard Purdie, on the review and validation of new contributions. Other Bootlin engineers have also helped in investigating and resolving specific bugs. We will also continue this involvement in the Yocto Project build engineering in 2022.

On the Buildroot side:

  • We have continued our work towards the implementation of top-level parallel build support in Buildroot. Bootlin engineer Hervé Codina has posted several iterations of a patch series bringing a mechanism preventing file overwrites between packages, a requirement for proper top-level parallel build.
  • We implemented and published, in partnership with ST, a Buildroot BR2_EXTERNAL called buildroot-external-st which contains example configurations for the STM32MP1 platforms from ST.
  • We contributed support for hybrid ISO9660 images, supporting in a single image PC platforms based on legacy BIOS, 32-bit UEFI BIOS and 64-bit UEFI BIOS, using the grub2 bootloader.
  • We helped one of our customers reduce their out-of-tree Buildroot patches by upstreaming a number of new Buildroot packages.

Outside of our the Yocto Project and Buildroot, we also ventured into the world of Debian/Ubuntu for embedded systems, by using the ELBE tool, to which we contributed support for building Ubuntu-based images. See our blog post that describes how to use ELBE to automatically build Ubuntu-based images for the Raspberry Pi.

Audio support in Linux

Audio support in the Linux kernel is another area where Bootlin engineers have specific expertise with. Here are some examples of audio related projects we worked on in 2021:

  • Integration of audio support for an i.MX6 platform with a complex dual AD1978 audio codec configuration, with a TDM8 audio interface between the SoC and the audio codecs.
  • Development of a brand new and complete ALSA driver for a new PCIe sound card, based on a FPGA.
  • Integration of audio support for an i.MX8 platform used in the automotive space with a complex audio configuration that involves multiple A2B transceivers through the Analog Devices AD2428 A2B master, Bluetooth audio, and more.

OP-TEE

In 2021, we started working on a major project: adding support for the Microchip SAMA5D2 processor to the OP-TEE project. This project is already well underway, as we have a functional port of OP-TEE, which is now being upstreamed.

As part of this, Bootlin engineer Clément Léger has implemented and contributed a generic clock framework for OP-TEE, which has already been accepted upstream. See our blog post for more details.

Clément also gave a talk at the Embedded Linux Conference 2021 on the topic of OP-TEE, titled OP-TEE: When Linux Loses Control. Slides and video are available.

We expect to continue the upstreaming of the SAMA5D2 support in 2022, and also develop support for additional SAM5D2 hardware capabilities in OP-TEE.

U-Boot extension board manager

In collaboration with the BeagleBoard.org community, we developed and contributed to U-Boot a generic extension board manager. This mechanism allows hardware platforms such as the BeagleBone to automatically detect extension boards that are connected, and apply the Device Tree overlays that provide the hardware description for those extension boards.

This work was covered in detail in the talk given by Bootlin engineer Köry Maincent at the Live Embedded Event in June 2021, see the slides and video.

Modems

In 2021, we had the opportunity to work on several projects that involved 3G/4G/5G modems, strengthening our knowledge of the modem stack in Linux, especially around modem-manager, libmbim and libqmi.

For example, we worked on the support of a Quectel BG95 modem interfaced with a RaspberryPi 4, or the support of a Sierra Wireless EM9190 modem interfaced over PCIe to an NXP i.MX6. The latter was particularly challenging and is still on-going, as the upstream Linux support for PCIe 5G modems is still very recent. We contributed a few fixes in this area.

Secure boot

Secure boot remains an important topic for a growing number of projects, and in 2021, we helped customers with secure boot on several i.MX6 platforms, and on one i.MX8 platform. The Yocto integration of secure boot was also a key aspects in those projects, to get a proper process for signing and verifying all states of the boot process. We also have a few U-Boot contributions in our contribution pipeline related to improving secure boot support. See for example the talk from Thomas Perrot at Live Embedded Event, relating the secure boot setup on i.MX8.

Camera, ISP and video support

We finished 2020 with a lot of on-going contributions to the Linux Video4Linux subsystem, which we had summarized in a blog post early 2021. In particular, our drivers for the OV5568 and OV8865 camera sensors were merged upstream.

In 2021, Bootlin engineer and multimedia expert Paul Kocialkowski continued to work on several multimedia topics. Some of the key projects included:

  • Optimizing the H264 decode → rescale → H264 encode pipeline of one of our customers, on an Allwinner H3 platform. We were able to significantly optimize the pipeline by leveraging the H264 decoder built into the Allwinner processor, for which Paul had written a Linux kernel driver several years ago, and by developing a custom ffmpeg plugin that offloaded the rescaling to the GPU, using the open-source lima support. A fully open-source solution!
  • Developing a proper Linux kernel driver for the Allwinner ISP, with support for debayering and noise filtering. This was a significant challenge as the Allwinner ISP was so far only supported through closed-source binary blobs. We have already submitted a first iteration to the upstream Linux kernel community. See our blog post for more details, as well as the talk given by Paul at the Embedded Linux Conference: slides and video.
  • Improving the Allwinner Linux kernel camera driver to support single buffer capture. This is useful in scenarios where camera sensors are not used to capture a sequence of frames, but just single frames, and the available memory is limited. This is particularly true on platforms such as the Allwinner V3s, which may be limited to just 64MB of RAM. This is going to be submitted upstream soon, as part of our on-going work on the Allwinner camera driver.

NAND and flash support

With Bootlin engineer Miquèl Raynal being a maintainer of the NAND subsystem in Linux and a co-maintainer of the MTD subsystem, it should be no surprise that we have continued to deal with a number of flash memory related projects in 2021:

  • We have brought in mainline the support for several NAND flash controllers:
    • For the Arasan NAND controller, used in some Xilinx processors. See our blog post on this topic.
    • For the ARM Primecell PL35x NAND controller, used in some other Xilinx processors. See our blog post on this topic.
    • For the NAND controller found at least in the the Renesas R-Car Gen3 and RZ/N1 processors.
  • Contributed support for the NV-DDR interface, which is used by some NAND flash chips and controllers to increase the throughput. See our blog post.
  • Continued our work on generalizing ECC support in the MTD subsystem, so that all ECC setups (software ECC, on-die ECC, ECC in the NAND controller, or external ECC engine) can be supported regardless of the NAND interface (serial or parallel). See the talk from Miquèl Raynal, ECC engines given at last year’s ELCE.
  • Improved the TI GPMC NAND controller driver to properly support NAND chip with large pages (larger than 4KB).

ADC/IIO support

We’ve helped a few customers with Linux kernel support for ADC devices, namely:

  • Extend the support for the MAX1027 family of ADCs, with support for external triggers
  • Bring support for the ADC found in TI AM437x processors

All of these were contributed to the upstream Linux kernel, Miquèl Raynal also wrote a extensive blog post on various aspects of the IIO subsystem and has taken the opportunity of these projects to also improve/clarify various aspects of the IIO core in a recent patch series.

Networking

Networking support in the Linux kernel is also one area where Bootlin is very active. Here are some key projects we worked on in 2021, some of them being on-going with additional work expected in 2022:

  • We implemented support for PTP offloading for the Qualcomm AR803X PHY.
  • We started working on QUSGMII support in Linux, a new standard that allows PTP time-stamps to be included directly on the preamble of QSGMII frames.
  • We started working on the 802.15.4 stack in the Linux kernel, with our initial step being support for passive scanning.
  • We implemented mqprio support in the mvneta Linux kernel driver, used for some Marvell platforms. mqprio is a queuing discipline that allows mapping traffic flows to hardware queue ranges using priorities and a configurable priority to traffic class mapping.
  • We implemented Frame DMA support in the Ocelot Ethernet switch driver, significantly improving the performance of frame injection/extraction by the CPU into/from the switch traffic.
  • We developed proper SFP support in Linux and U-Boot, with dynamic reconfiguration (in Linux only), for a customer using the Zynq 7000 platform and a complex network setup.

BSP development

Many of our projects are obviously related to the development of update of complete Linux BSP for our customers (bootloader, Linux kernel, custom embedded Linux distribution). Here are a few examples:

  • Development of a brand new U-Boot, Linux and Yocto based BSP to migrate an existing product running Windows CE on a ST Spear320 processor to Linux. This sort of development proved to be challenging as the Spear320 support in both U-Boot and Linux is close to be abandoned (and in fact has since then been removed from U-Boot upstream). We hope to be able to contribute to improve the upstream Spear320 support in 2022.
  • Migration of an AM335x/Buildroot based BSP to newer Linux kernel and Buildroot releases, for a customer in the healthcare industry
  • On-going migration of an i.MX6 BSP from an old kernel release and a custom build-system to recent versions of U-Boot and Linux as well as a proper standardized Yocto Project based Linux distribution. This project presents some interesting challenges as it uses only one of the two Cortex-A9 cores to run Linux, the other one runs a bare-metal application, and we will also migrate this to a proper usage of the remoteproc and rpmsg mechanisms.
  • We worked with several customers on STM32MP1 platforms, helping with porting on new platforms, extending the Device Tree and device drivers, developing custom Yocto-based or Buildroot-based distributions. STM32MP1 is definitely becoming a popular platform for a number of new projects.
  • We also worked with several customers on custom platforms based on the RasberryPi Compute Module, doing Device Tree configuration/tweaking and Yocto integration.

Over-the-Air update integration

We continued to help our customers with the integration of Over-The-Air update solutions in their embedded Linux systems. A few examples:

  • We integrated RAUC for an i.MX6 platform, using the Barebox bootloader and Yocto Project based distribution. Bootlin engineer Kamel Bouhara wrote a detailed blog post on this topic, as well as the talk from Kamel at Live Embedded Event
  • We integrated the Azure Device Update for IoT Hub, using the U-Boot bootloader on a RaspberryPi CM4 platform, also with a Yocto Project based distribution. Internally, the Device Update for IoT Hub is based on swupdate
  • As part of a migration of an existing embedded product based on Debian to a Yocto-based distribution, we are integrating Mender.

Crédit Impôt Recherche

For our French customers, another important milestone achieved by Bootlin in 2021 is the delivery of our Crédit Impôt Recherche agreement, which allows our French customers to benefit from tax incentives on research and development activities done by Bootlin for their projects. See our blog post for more details.

Training

With the COVID19 still making travel conditions difficult and uncertain, most of our training activity in 2021 was dedicated to on-line training courses. Indeed, 69 out of our 72 training sessions were delivered online this year. In total, we delivered our training courses to 906 engineers in 2021.

In 2021, we published two new training courses:

Just like all our other training courses, the complete training materials for those new courses are freely available, distributed under the CC-BY-SA license.

Another major event in 2021 was our work to get the French Qualiopi certification, which proves the quality of our training organization and processes. As part of this, we have improved several aspects of our training courses, mainly regarding feedback collection and handling as well as the evaluation of the participants.

In 2022, we expect to at least:

  • Improve how our online course are delivered, by ensuring more participants can do the practical labs by themselves, which is an important part of the learning process
  • Publish at least one more training course. We already have plans for a course on a topic that we think will be very relevant to many embedded Linux engineers. Stay tuned for updates on this!
  • Have additional capacity to deliver our training courses, which are seeing significant demand.

For more details about our training offering, see our training page, which details our dedicated on-line and on-site sessions as well as our public on-line sessions.

Contributions

In Bootlin continued and strong open-source focus, we once again contributed to different open-source projects in 2021:

  • Linux kernel. The best summaries are our blog posts about our contributions to Linux 5.10, Linux 5.11, Linux 5.12, Linux 5.13, as well as Linux 5.14 and 5.15. Some of our significant contributions: huge effort on ECC engine support in the MTD subsystem, rv3032 RTC driver, support for new MIPS platforms from Microchip, a Simple Audio Mux driver, major work on timer/TCB support on Microchip ARM platforms, new I3C master controller driver for the Silvaco I3C IP, new drivers for the OV5568 and OV8865 camera sensors, NAND controller drivers for the Arasan, PL35x and Renesas IPs, enabling of ADC support on TI AM437x.
  • U-Boot. Our biggest contribution has been the extension board manager, which is described earlier in this blog post. We also contributed various small fixes and improvements.
  • OP-TEE. We contributed a generic clock framework, and the support for the SAMA5D2 clock tree and TRNG driver. So far 55 commits from Bootlin have been integrated in OP-TEE.
  • Buildroot. We contributed a total of 257 patches. The main contributions are in the area of tooling to list security vulnerabilities in Buildroot packages, support for Bootlin external toolchains, improved SELinux support, support for the Beagle-V RISC-V platform, a good number of new packages, support for hybrid ISO9660 images. In addition, Thomas Petazzoni continued his work as a co-maintainer of the project: Thomas has reviewed and merged close to 2000 patches from Buildroot contributors throughout 2021.
  • Yocto Project. As outlined above in this blog, we did major contributions to the Yocto Project, with Michael Opdenacker becoming co-maintainer of the Yocto Project documentation, and Alexandre Belloni being involved in the build and release engineering effort.
  • Linux Test Project. We fixed a number of issues in LTP tests that prevented from using LTP in embedded-oriented systems, such as the ones that can be generated using the Yocto Project. See our blog post for more details.
  • Our training materials, which are all freely available, have seen no less than 872 commits. These includes updates to our existing training courses, but also our new Real-Time Linux with PREEMPT_RT training course.
  • Our freely available toolchains have also been improved and updated. See our blog post.
  • We continued to maintain our very popular Elixir code indexing tool.

Conferences

Here are the talks that we presented this year, at various virtual events:

  • FOSDEM 2021
    • Networking Performance in the Linux Kernel, Getting the most out of the Hardware, by Maxime Chevallier. Slides and video.
    • Embedded Linux from scratch in 45 minutes, on RISC-V, by Michael Opdenacker. slides and video.
  • Live Embedded Event
    • Security vulnerability tracking tools in Buildroot, by Thomas Petazzoni. Slides and video.
    • Secure boot in embedded Linux systems, by Thomas Perrot. Slides and video.
    • Understanding U-Boot Falcon Mode, by Michael Opdenacker. Slides and video.
    • Device Tree overlays and U-Boot extension board management, by Köry Maincent. Slides and video.
    • Getting started with RAUC, by Kamel Bouhara. Slides and video.
  • Embedded Linux Conference
    • I3C in tomorrow’s design, by Miquèl Raynal. Slides and video.
    • Embedded Linux nuggets found in Buildroot package Eldorado, by Michael Opdenacker. Slides and video.
    • OP-TEE: When Linux Loses Control, by Clément Léger. Slides and video.
    • Advanced Camera Support on Allwinner SoCs with Mainline Linux, by Paul Kocialkowski. Slides and video.
  • A webinar organized in partnership with ST, Device Tree 101, by Thomas Petazzoni. Slides and video.
  • A webinar organized in partnership with Microchip, Improving Linux Boot Time in Your Embedded Application, with the participation of Thomas Petazzoni. Link.