Linux 6.0 released, Bootlin contributions

Linux 6.0 has been released two weeks ago, and Linux 6.1-rc1 is already out of the door, but we didn’t get the chance to look at the contributions made by Bootlin to the Linux 6.0 release. Before we do that, let’s provide our usual must-read articles on Linux 6.0: the Linux 6.0 merge window part 1 and Linux 6.0 merge window part 2 LWN.net articles and the KernelNewbies.org article.

On Bootlin side, our significant contributions to this release have been:

  • Clément Léger contributed a new driver for the Ethernet switch found in the Renesas RZ/N1 processor, as well as a PCS driver for the MII converter of the same processor. Obviously, this came with the related Device Tree bindings and Device Tree changes, but also with a few small changes in the DSA subsystem.
  • Hervé Codina enabled support for the PCIe controller found in the same Renesas RZ/N1 processor, which in fact does not allow to use PCIe devices, but USB devices: this PCIe controller is only used to connect to an internal USB controller in the chip, which therefore allows to use USB devices.
  • Köry Maincent extended the existing mpc4922 DAC IIO driver to also support the mpc4921 variant, which has only one output channel instead of two.
  • Luca Ceresoli contributed several improvements to the I2C subsystem documentation.
  • Paul Kocialkowski contributed a new DRM driver for the logiCVC-ML display controller IP
  • Paul Kocialkowski contributed two new V4L drivers for the MIPI CSI-2 camera interfaces available in the Allwinner A31 family of processors (sun6i) and the Allwinner A83T family of processors (sun8i).

Here is the full details of our contributions, commit by commit:

Improved version of our popular “Embedded Linux system development” course

Embedded Linux system development courseOur Embedded Linux system development course has been for many years one of our most popular training courses. It is our course targeted at engineers who are getting started with Linux on embedded systems, and need to understand the big picture, but with a sufficiently deep level of details. It describes the overall structure of an embedded Linux system, and teaches step by step how it is build: cross-compilation toolchain, bootloader, Linux kernel, minimal root filesystem, storage, integration of user-space components, build systems, etc.

Even though this course has seen many small updates throughout the years to use newer versions of the different software components and various improvements and updates to the training materials, the course had remained fundamentally the same for quite some time. However, we had identified a number of areas on which we wanted to make some more fundamental changes, and we’ve taken the chance of the summer season to prepare a brand new version of this course, with many major changes and improvements.

This updated version of the course is now available, and as usual with Bootlin its training materials are freely available:

Embedded Linux system development courseCompared to our previous version of this course, the main changes are:

  • Major rewrite of the lecture section on bootloader and firmware, to better cover UEFI, Trusted Firmware (TF-A), Trusted Execution Environment, and overall reflect the increased complexity of the booting process of modern embedded platforms. The corresponding practical lab also makes use of TF-A to illustrate this.
  • Addition of a new section on Accessing hardware devices with many details on the Device Tree, how to identify kernel drivers for devices, and what are the typical interfaces in Linux to access hardware. This is illustrated by a new lab in which we manipulate GPIOs, LEDs, add support for a sound card connected over USB, and add support for a joystick connected over I2C, which extending the Device Tree and manipulating pin-muxing.
  • Removal of the practical lab on flash filesystems, due to the progressively reducing number of platforms that use raw NAND flash. We still have a lecture on how raw flash memory is handled in Linux (MTD, UBI, UBIFS), but no longer a practical lab, in order to spend time on topics that are more commonly relevant.
  • A major rework of the final part of the course which covers the user-space stack, with the aim of showing how to build a reasonably realistic product:
    • We explain and demonstrate how to cross-compile and integrate manually libraries and applications in an embedded Linux system. We illustrate this by cross-compiling alsa-lib and alsa-utils to play audio with our USB audio device, libgpiod to manipulate GPIOs, and ipcalc to manipulate the Meson build system.
    • We then explain the concepts and principles behind embedded Linux build systems (with a focus on Buildroot and Yocto), as well as binary distributions. We illustrate this by using Buildroot to build a complete embedded Linux system, which uses mpd as an audio player daemon.
    • We then cover open-source licensing topics.
    • We then cover the major Linux software stacks for graphics, multimedia, networking, as well as systemd and D-Bus. We illustrate this by changing our system to use systemd as an init system, and use udev for device management.
    • Finally, we cover application development and debugging: how to cross-compile your own application, how to debug it, using strace, ltrace, gdb, perf, valgrind. We illustrate this by implementing an application that allows to use an I2C-connected joystick to control the audio playback of an audio playlist. This application is then analyzed and debugged using the relevant debugging tools.
  • The real-time Linux part of the course has been removed from the course, as we now have a dedicated real-time Linux with PREEMPT_RT course, which goes into many more details on this topic.

The course is currently delivered with practical labs done on the STM32MP1 platform from ST, but we intend to port it on the BeagleBone Black and Qemu as well. In any case, the course is very generic and relevant for all embedded Linux projects, regardless of the specific hardware platform being used.

This new edition has already been delivered to several customers, both on-line and on-site. All our slots for 2022 are already fully booked, but we do offer for 2023:

  • A public on-line session on January 30, 31, February 2, 3, 6, 7, 9, 2023, from 2:00 PM to 6:00 PM UTC+1. Registration is accessible directly online. This is the best option to train a few engineers from your team.
  • Private sessions, either on-line or on-site, upon request. You can contact us to discuss the details and get a quote.

We are really looking forward to continuing to share our knowledge about embedded Linux with even more engineers, and are confident that this updated version of the course will make this knowledge sharing even more efficient and fruitful.

Bootlin toolchains 2022.08 released

Bootlin toolchains 2022.08The toolchains.bootlin.com service provides freely available pre-compiled cross-compilation toolchains for a wide range of CPU architectures and configurations.

We have just published version 2022.08 of these toolchains, which are now built using Buildroot 2022.08. Thanks to this the toolchains now use the following components:

  • The bleeding-edge toolchains are based on gcc 12, binutils 2.39, gdb 12, kernel headers 5.4, glibc 2.35 or musl 1.2.3 or uclibc-ng 1.0.42.
  • The stable toolchains are based on gcc 11, binutils 2.38, gdb 11, kernel headers 4.9, glibc 2.35 or musl 1.2.3 or uclibc-ng 1.0.42.

Even though glibc 2.36 has been recently released, it still caused too many issues to be integrated in this toolchain release. glibc 2.36 will be part of a future update of these toolchains.

We also have a few new toolchains that appeared:

  • Toolchains for the OpenRISC CPU architecture, based on the glibc C library. We already had toolchains for OpenRISC, but not using glibc.
  • We now have both stable and bleeding-edge toolchains for the x86-64-v2, x86-64-v3 and x86-64-v4 architecture variants. We used to have only bleeding-edge toolchains for these variants as only the latest gcc had support for them.

The only toolchain that was not updated as part of this release is the m68k Coldfire toolchain, as we currently have a regression on elf2flt. This will hopefully be adressed in the future.

If you have feedback or encounter any issue in using these toolchains, the project issue tracker is where you should go.

Welcome to Théo Lebrun!

Welcome on board!We are happy to announce that Théo Lebrun has joined our engineering team, as of September 9, 2022, just in time to attend the Embedded Linux Conference Europe with the rest of the Bootlin engineering team.

Théo has just graduated from the Université de Technologie de Belfort Montbéliard, and did his final 6-month internship at Bootlin, where he worked on PipeWire and support for the A2B Audio Bus in the Linux kernel.

Théo is joining our team based in Lyon, where Alexandre Belloni, Grégory Clement and Kamel Bouhara are based.

Welcome on board Théo!

Bootlin at Linux Plumbers conference 2022

Next week, almost the entire Bootlin team will be at the Embedded Linux Conference Europe in Dublin, see our previous blog post on this topic. We will give four talks at this event, on a variety of Linux kernel and embedded Linux topics.

During the same week, also in Dublin albeit in a different location, will take place the Linux Plumbers conference. Bootlin engineer Miquèl Raynal will give a talk at Linux Plumbers, as part of the IoTs a 4-Letter Word micro-conference. Miquèl’s talk will discuss Linux IEEE 802.15.4 MLME improvements, as Miquèl has been working for several months on bringing improvements to the 802.15.4 stack in the Linux kernel.

Buildroot Summer 2022 Hackathon

Buildroot is an easy-to-use and popular embedded Linux build system, used by many as an alternative to Yocto/OpenEmbedded. Bootlin has expertise in both build systems, and has in particular been a long time contributor to the Buildroot project. Bootlin CEO’s Thomas Petazzoni is one of the co-maintainers of the project, to which he has contributed over 5000 patches.

From July 23 to July 27, four members of the Buildroot community gathered in the sunny south of France for a 5-day long hackathon on Buildroot: Yann Morin, Romain Naour from Smile, Arnout Vandecappelle from Mind and Thomas Petazzoni from Bootlin.

The main goal of this hackathon was to reduce the backlog of patches accumulated in the project’s patchwork, the tool used in the Buildroot community to record all contributed patches and make sure all of them are handled: reviewed, accepted, or potentially rejected.

When we started the hackathon, the backlog of patches cumulated to over 410 patches, and the hackathon allowed to reduce the backlog to 150-160 patches, especially taking care of all patches contributed before the beginning of 2022. In total, over 400 patches were merged during the hackathon in the Buildroot Git repository, which obviously is more than the reduction of the backlog of patches. This is mainly due to additional patches contributed during the hackathon itself (the community has been very active at submitting new patches!) and due to the review process triggering additional ideas or rework that required more patches.

Here is a summary of the main highlights that were merged during this hackathon:

  • New package for ntpsec, a “a secure, hardened, and improved implementation of Network Time Protocol”
  • Some cleanup of the genimage configuration files to use shortcuts for common GPT partition GUIDs
  • Test cases added for the get-developers script
  • Addition of several new defconfigs for multiple RISC-V 64-bit noMMU platforms: Sipeed MAIX-Bit, Sipeed MAIXDUINO, Sipeed MAIX-Dock, Sipeed MAIX-Go and Canaan KD233, based on Canaan K210 SoC.
  • Migration of the .py to .pyc byte-compilation to use the built-in Python infrastructure instead of a custom script.
  • Addition of support for version 12 of the GCC compiler. This means that GCC 11 is now the default in Buildroot, and support for GCC 9 has been removed.
  • Many new Python packages added: python-lark, python-typeguard, python-mypy-extensions, python-typing-inspect, python-rfc3987, python-ruamel-yaml, python-pyrsistent, python-pylibfdt, python-maturin
  • Rust has been bumped to 1.62.0
  • New package for mender-connect, a daemon responsible for handling bidirectional (websocket) communication with the Mender server
  • Some fixes to the glslsandbox-player package
  • New package for OpenSC, a set of libraries and utilities to work with smart cards.
  • New package for hyperfine, a benchmark tool written in Rust. It evaluates execution time of a command passed in arguments and make a relative comparison if multiple arguments are used at the same time.
  • New package for hawktracer, a highly portable, low-overhead, configurable profiling tool built in Amazon Video for getting performance metrics from low-end devices.
  • New package for vis-network, a JS library to display dynamic, automatically organised, customizable network views
  • New package for Avocado, an automated testing suite containing tests for various subsystems.
  • New package for freeradius-server, an open source server which implements a protocol for remote user Authorization, Authentication and Accounting.
  • New package for volk, a Vector-Optimized Library of Kernels. It is a library that contains kernels of hand-written SIMD code for different mathematical operations.
  • A rework of the LTO handling: LTO support is now always enabled in GCC, and the new option BR2_ENABLE_LTO allows to request LTO to be used by packages that support it.
  • New package for the Qt 6 library, separate from the current Qt 5 package. For now, the Qt 6 package is very minimal: it only packages qt6base, and only the core of Qt, not even the GUI support is enabled. As part of this, the double-conversion and libb2 packages were added.
  • Support for configurable page size for ARM64 was added. In addition to the default 4 KB, 64 KB pages are supported. This includes some test cases that allow to validate the 64 KB ARM64 page size support in Qemu.
  • New package for clpeak, a tool that profiles OpenCL devices to find their peak capacities, together with its dependency OpenCL-CLHPP, the C++ bindings for OpenCL
  • A rework of several top-level menuconfig options: the “enable MMU” option is now part of the Target architecture menu, the Toolchain menu is now before the Build options menu, which allows the choice of the C library to be done before the choice of static vs. shared libraries. This allows the choice of static library to be made unavailable when glibc is selected, fixing a number of invalid configurations.
  • New package for GDAL, a translator library for raster and vector geospatial data formats.
  • New package for libutp, a uTorrent Transport Protocol library
  • New package for dust, an alternative written in Rust of the command “du”.
  • The C-SKY CPU architecture support was removed, as it was no longer maintained, and barely used.
  • New package for gitlab-runner, the open source project that is used to run your jobs and send the results back to GitLab
  • New package for dbus-broker, which is an alternative implementation of the D-Bus daemon, by the systemd community. It is integrated in Buildroot so that either classic D-Bus or dbus-broker can be used as the D-Bus daemon implementation.
  • New package for nerdctl, a Docker-compatible CLI for containerd, controlling runc.
  • A rework of how the udev hwdb is handled, to be consistent between systemd and eudev, and to remove useless hwdb source files from the target, as the hwdb is compiled at build time.
  • And at least 137 packages have seen their version bumped

Several other topics were looked at and discussed, but did not necessarily lead to patches already integrated. One such topic is the investigation of several issues with elf2flt, the tool used on noMMU architectures to produce binaries in the FLAT format, from an ELF binary. Another topic is the merge of the SciPy package, for which the review and testing is well advanced.

Overall, it was a very productive hackathon, and besides the massive work done on Buildroot from 9 AM to (at least) midnight each day, the participants also enjoyed lots of side discussions, embedded Linux related or not. We look forward to the next in-person gathering of the Buildroot community, on September 17/18 in Dublin, right after the Embedded Linux Conference Europe.

Linux 5.19 released, Bootlin contributions inside

Linux 5.19 has been released yesterday. We recommend the usual resources of LWN (part 1 and part 2) as well as KernelNewbies to get some high-level overview of the major additions. CNX-Software also has an article focused on the ARM/RISC-V/MIPS improvements.

At Bootlin, we contributed 68 patches to this release, the main highlights being:

  • Clément Léger contributed patches for the Microchip SAMA5 platform to support suspend operation while running in non-secure mode, with OP-TEE handling the necessary PCSI calls. This is related to our work to port OP-TEE on Microchip SAMA5D2, which we have covered in several blog posts before.
  • Hervé Codina contributed device Tree updates to enable the PCI controller of the Renesas RZ/N1 platform, which allows to access the USB host controller that sits on an internal PCI bus. Some driver updates for the PCI driver are needed, and they will land in 5.206.0 kernel.
  • Miquèl Raynal contributed several improvements to the IIO subsystem, following his work on several IIO drivers and his related blog post. These improvements either touch the core IIO, or fix some incorrect API use in IIO drivers.
  • Miquèl Raynal contributed a new driver for the Renesas RZ/N1 DMA router (in drivers/dmaengine) as well as a new driver for the Renesas RZ/N1 Real Time Clock (in drivers/rtc). In addition, Miquèl modified the 8250 UART controller driver to be able to use the DMA capabilities available on the RZ/N1 processor.
  • Miquèl Raynal also contributed a number of improvements to the IEEE 802.15.4 stack in the Linux kernel.
  • Paul Kocialkowski contributed support for MIPI CSI-2 in the Allwinner phy-sun6i-mipi-dphy driver.
  • Paul Kocialkowski and Luca Ceresoli contributed a few misc fixes, touching the SPI core and SPI Rockchip driver and the dmaengine documentation.

The complete details of our contributions are:

Yocto training course updated to Kirkstone release

Yocto ProjectBack in May 2022, the Yocto Project published the Kirkstone release, the latest Long Term Support version of the popular embedded Linux build system. See the release notes of this 4.0 release.

For many years, Bootlin has helped plenty of engineers around the world get started with the Yocto Project and OpenEmbedded thanks to our Yocto Project and OpenEmbedded development training course, whose training materials are freely accessible, to everyone.

We are happy to announce that we have just published the update of these materials to cover this new Kirstone release. Our practical lab instructions, available for both the STM32MP1 Discovery Kit and the BeagleBoneBlack, have been correspondingly updated:

This update was done by Bootlin engineer Luca Ceresoli, who is teaching this updated version of our course this week. The next public session, open to individual registration, will take place on September 26-30, and will be taught by Maxime Chevallier. If you are interested, you can register directly online. We also offer this course in private sessions, organized on demand for your team, either on-line or on-site, you can contact us for more details.

Embedded Linux Conference Europe 2022: four talks from Bootlin

The schedule for the upcoming Embedded Linux Conference Europe 2022 has been published recently.

Bootlin CEO Thomas Petazzoni is again a member of the program committee for this edition of ELCE, and has helped with other members of this committee in reviewing and selecting the numerous talk submissions that have been received.

Bootlin will obviously be present at this conference. With 13 engineers from Bootlin participating, almost the entire company will be in Dublin for this major event of the embedded Linux community. Also, 4 of the talks that we had submitted have been accepted:

  • Luca Ceresoli on Basics of I2C on Linux
    This talk is an introduction to using I²C on embedded Linux devices. I²C (or I2C) is a simple but flexible electronic bus to allow low-speed communication between the CPU and all sorts of chips: PMICs, ADC/DACs, GPIO expanders, video sensors, audio codecs, EEPROMS, RTCs and many more. It is so popular that knowing it is a must for any embedded system engineer. Luca will first give an introduction to what I2C is at the electrical level. He will then describe how I2C is implemented in the Linux kernel driver model and how that appears in sysfs, how to describe I2C devices using device tree and how to write a driver for an I2C device. Finally he will present the tools to communicate with the chips from userspace and share some debugging techniques, including inspection of the physical bus and software-level debugging.
  • Miquèl Raynal on Improving Wireless PAN Support
    Anybody eager to learn about IoT devices has at least once tried to play with Zigbee or 6lowpan sensors. These two protocols are built on top of a well common MAC/PHY specification: IEEE 802.15.4, also known as Wireless Personal Area Networks: WPAN, designed to be low-rate/low-range wireless networks. There is already substantial support for this protocol in the Linux kernel but when my journey started, several of the MAC-related operations well described in the specification were not implemented, making the subsystem mainly useful for very simple use cases: peer-to-peer transmissions. This is unfortunate as a significant part of the idea behind WPAN is to make these networks quite adaptive and resilient, which requires a minimal subset of the peer management procedure to be supported. Besides a number of preparation changes, the main idea behind the continuous flow of patches was to bring support for the scanning procedure which allows a PAN controller to detect all the compatible devices around it in different ways. Discovering these devices is the first step in order to associate them together and build up starred networks. This talk will be an opportunity to explain the new APIs allowing such discoveries and provide a state of the art of the support in the mainline kernel.
  • Michael Opdenacker on Implementing A/B System Updates with U-Boot
    A popular way to implement system updates is through the A/B scheme, in which you have two copies of the root filesystem, one which is active, and one that is meant to contain the next update. When a new update is successfully applied, you need to make the corresponding partition become the new active one. That’s when a number of practical questions arise, such as how to identify the active partition, how to detect when the new system fails to boot properly, and how to fall back to the previous version? It was hard to find documentation about how U-Boot could address such needs to implement a functional and failsafe A/B system update mechanism. This presentation proposes to address this need by sharing the practical solutions we found, using lesser known commands and capabilities in U-Boot. We will also explain how the Linux side can cooperate with the U-Boot side. Fortunately, you won’t need to erase half of your brain to get updated on this topic.
  • Paul Kocialkowski on Walking Through the Linux-Based Graphics Stack
    The graphics stack used with the Linux kernel is a notoriously complex beast. From userspace down to the kernel level, a number of components are involved and interact with eachother. It is also an area that is constantly evolving to meet new use cases, refresh legacy implementations and achieve better performance. This makes it difficult to have a clear idea of the big picture and what is actually happening when using graphics-related components. This presentation will detail a walk through the graphics stack, with actual examples of displaying a buffer and rendering using the GPU. Going from the application level through the system libraries, down to the kernel and ending with actual hardware configuration. State-of-the-art technologies such as Wayland and DRM will be highlighted with relevant excerpts from the source code of related free software projects that are widely used today.

We look forward to meeting again the embedded Linux community, its developers, users and maintainers, at Dublin during this conference!

Updated Buildroot support for STM32MP1 platforms

Back in December 2021, we had announced the buildroot-external-st project, which is an extension of the Buildroot build system with ready-to-use configurations for the STMicroelectronics STM32MP1 platforms.

More specifically, this project is a BR2_EXTERNAL repository for Buildroot, with a number of defconfigs that allows to quickly build embedded Linux systems for the STM32MP1 Discovery Kit platforms. It’s a great way to get started with Buildroot on those platforms.

Today, we are happy to announce an updated version of this project, published under the branch st/2022.02 at https://github.com/bootlin/buildroot-external-st. This new version brings the following changes:

  • Updated to work with Buildroot 2022.02, the current LTS version of Buildroot
  • Updated to use the 4.0 “ecosystem” from ST, which means we’re using updated BSP components from ST, namely Linux 5.15, U-Boot 2021.10, TF-A 2.6 and OP-TEE 3.16
  • New defconfigs have been added to support all variants of the STM32MP157 Discovery Kits: STM32MP157A-DK1 and STM32MP157D-DK1, as well as STM32MP157C-DK2 and STM32MP157F-DK2.
  • The minimal defconfigs now use OP-TEE as BL32 instead of the minimal monitor provided by TF-A, called SP-MIN
  • The minimal defconfigs now have mdev enabled, to benefit from automatic kernel module loading
  • The demo defconfigs now have the Dropbear SSH server enabled

The document available on the Github page details how to use this work, but here is a quick start in just a few steps:

  1. Retrieve Buildroot itself, a branch containing a few patches on top of upstream 2022.02 is needed
    $ git clone -b st/2022.02 https://github.com/bootlin/buildroot.git
  2. Retrieve buildroot-external-st
    $ git clone -b st/2022.02 https://github.com/bootlin/buildroot-external-st.git
  3. Go into the Buildroot directory
    $ cd buildroot/
  4. Configure Buildroot, for example here the demo configuration for the STM32MP157F-DK2
    $ make BR2_EXTERNAL=../buildroot-external-st st_stm32mp157f_dk2_demo_defconfig
  5. Run the build
    $ make
  6. Flash the resulting SD card image available at output/images/sdcard.img and boot your board!

If you have any question or issue, feel free to use the Github issue tracker to contact us. Bootlin is an ST Authorized Partner, and can provide engineering and training services around embedded Linux on STM32MP1 platforms.