Bootlin toolchains 2023.11 released

Bootlin toolchains 2023.11Our toolchains.bootlin.com project offers a wide range of freely available pre-compiled cross-compilation toolchains, updated on a regular basis.

We are happy to announce the availability of version 2023.11 of our toolchains. Even though they are released in December (so one would expect the version number to be 2023.12), they are all generated using Buildroot 2023.11, hence the release number used for the toolchains.

Compared to the previous 2023.08 toolchains, here are the relevant changes:

  • Bleeding edge toolchains are now using gdb 14.1 instead of 13.2, glibc 2.38 instead of 2.37 and uclibc 1.0.45 instead of 1.0.43. They continue to use gcc 13.2 and binutils 2.41, as well as musl 1.2.4
  • Stable toolchains are now using gdb 13.2 instead of 12.1, glibc 2.38 instead of 2.37 and uclibc 1.0.45 instead of 1.0.43. They continue to use gcc 12.3 and binutils 2.40, as well as musl 1.2.4
  • The glibc version used has the fix for CVE-2023-4911
  • The gdb build has been fixed to no longer rely on uninstalled libbfd.so and libopcodes.so libraries
  • The zlib library, which was incorrectly present in the toolchain sysroot, is gone, fixing various build failures encountered with 2023.08 toolchains
  • There are now toolchains for m68k 68xxx based on uclibc and musl in addition to glibc, which was already supported

We have already submitted a patch to update the support of those toolchains in Buildroot, where they can be used as external toolchains.

Hopefully these toolchains will continue to be useful to the embedded Linux community. For any question, feedback or issue, you can use the Github issue tracker of the project.

Bootlin establishes a strategic partnership with Ratiotech, an expert in electronics and hardware design

For close to 20 years, Bootlin has offered to companies around the world its expertise in embedded Linux system development, with engineering and training services covering low-level software development in the Linux kernel and open-source bootloaders, as well as embedded Linux system integration and build systems.

Bootlin’s expertise is obviously tightly coupled with hardware engineering, as the low-level software we develop runs on a wide variety of hardware platforms, sometimes with complex designs. As experts in low-level software, our team possesses a robust understanding of the hardware used in embedded systems, including SoCs, board design, hardware interfaces, and protocols.

Today, we are delighted to announce a strategic partnership with Ratiotech, marking a significant expansion of our collective expertise. Ratiotech brings an impressive 25 years of experience in hardware design, electronic systems, signal integrity, and EMC.

Bootlin / Ratiotech partnership

This partnership enables:

  • Bootlin to offer significantly deeper and broader expertise in investigating, debugging, and resolving issues during the bring-up of our customers’ hardware platforms
  • Ratiotech to leverage Bootlin’s solid expertise in embedded Linux development, providing customers with a comprehensive range of services covering both hardware development and Linux operating system integration
  • A collaboration that allows Bootlin and Ratiotech to provide turn-key solutions, encompassing the entire development of hardware/software solutions: hardware design, industrialization, low-level software development on microcontrollers, porting and integration of embedded Linux systems, and application development

Commencing in early 2024, Bootlin and Ratiotech will share offices near Toulouse, France, facilitating a seamless collaboration between the teams of both companies to ensure the success of this partnership.

Ratiotech’s CEO, Fabien Hue, states, “This strategic partnership is an excellent step to accelerate the development of Ratiotech and offer our customers a broader expertise, fully complementing Bootlin’s expertise in embedded Linux systems.”

Thomas Petazzoni, Bootlin’s CEO, adds, “The expertise and offerings provided by Bootlin to its customers will be significantly enhanced through this partnership with Ratiotech. This close relationship will not only expedite the bring-up of new hardware platforms but also enable us to offer essential hardware development, debugging, and investigation expertise that many of our customers require. We are eagerly looking forward to further assisting our customers with Ratiotech’s expertise.”

For more details on this partnership, check our hardware expertise, and for more details about Ratiotech, visit their website at Ratiotech.

Keep your Yocto layer simple! Introducing meta-kiss, a working reference Yocto/OE setup

At Bootlin we help many of our customers using Yocto/OpenEmbedded to build the Linux software stack running on their end products. While doing that we have seen all sorts of problems caused by all sorts of complicated code in their build system setup. So we wondered what we could do to improve the overall situation. Continue reading “Keep your Yocto layer simple! Introducing meta-kiss, a working reference Yocto/OE setup”

Open-source Linux kernel support for the Allwinner V3/V3s/S3 H.264 video encoder

Bootlin has been involved with improving multimedia support on Allwinner platforms in the upstream Linux kernel for many years. This includes notable contributions such as hardware-accelerated video decoding initial support for MPEG-2, H.264 and H.265 following the successful crowd-funding campaign in 2018, support for the MIPI CSI-2 camera interface and early support for the Allwinner V3/V3s/S3 Image Signal Processor (ISP) in mainline Linux.

In addition to this work focused on Allwinner platform, we have also developed and released in 2021 initial Linux kernel support for the Hantro H1 H.264 stateless video encoder, used on Rockchip processors, on top of the mainline verisilicon/hantro driver.

The Allwinner V3s chip on the Lichee Pi Zero

Today Bootlin is happy to announce the release of Linux kernel support for H.264 video encoding with the Allwinner V3/V3s/S3 platforms, in the form of a series of patches on top of the mainline Linux cedrus driver (which already supports decoding) and a dedicated userspace test tool. The code is available in the following repositories:

The updated cedrus media controller graph.

This work is both the result of our internal research and a continuation of earlier projects that were carried out by members of the linux-sunxi community to document and implement early proofs of concepts for H.264 encoding, covering older platforms such as the A20 and H3. We would like to give a warm thank-you for these previous contributions that made life significantly easier for us.

Adding support for H.264 encoding required bringing a significant architecture rework to the driver, which was rather messy and not very well structured (but it was of course quite difficult for us to find the most elegant way of writing the driver back in 2018, when we were just getting start on the topic). With a clear organization and the adequate abstraction in place, it became much easier to add encoding support and focus on the hardware-specific aspects.

Just like decoding, encoding is based on the V4L2 M2M framework and shares the same V4L2 and M2M devices with decoding (meaning that only a single job of either decode or encode can be processed at a time). Two video device nodes are exposed to userspace, allowing as many decoding and encoding contexts as needed to exist simultaneously. Advanced H.264 codec features such as CABAC entropy coding and P frames are supported.

However this work is not yet suitable for inclusion in the mainline Linux kernel since there is no well-defined userspace API (uAPI) for exposing stateless encoders. We have started discussions to converge towards an agreeable proposal that would be both generic enough to avoid device-specific considerations in userspace while also benefiting from all the features and specificities that stateless encoders can provide. While discussions are still in progress, others have expressed interest in the same topic for the VP8 codec. Our previous work published in 2021 for the Hantro H1 H.264 stateless video encoder is also not upstream yet for the same reason: the need for a well-defined userspace API to expose stateless encoders.

Besides the introduction of a relevant new uAPI, a number of challenges remain in the path towards full support for Allwinner H.264 video encoding in mainline Linux:

  • The rework of the driver needs to be submitted and merged upstream;
  • Rate-control is currently not implemented and only direct QP controls are available;
  • This work only covers the Allwinner V3/V3s/S3 platforms, while most other generations also feature different (yet rather similar) H.264 encoder units that could also be supported with some dedicated effort;
  • Pre-processing features such as scaling and pixel format conversion are not yet support;
  • Developing userspace library support (such as FFmpeg or GStreamer) to make use of the stateless encoder uAPI would be necessary;

If you are interested in funding the effort to help us advance any of these topics, feel free to get in touch with us to start the discussion!

Back from Netdev 0x17

At Bootlin, we focus on Embedded Linux development and support, and these embedded devices often have a network interface, be-it an Ethernet port, a Wireless chip or some other kind of communication channel that falls under the Linux Networking Stack’s framework.

So it’s always interesting to see what the rest of the community is working on, and meet in real life people we interact with on the netdev mailing list.

That’s why this year, Alexis LothorĂ© and Maxime Chevallier flew to Vancouver to participate to the Netdev Conference, a 5 days event organised by the Netdev Society, a small non-profit run by volunteers dedicated to holding this event.

Bootlin at Netdev

Most talks at Netdev are not directly covering topics we’re actively working on, but it’s always refreshing to see these new exciting technologies that could trickle their way down to the embedded world a few years from now. It is also always pretty interesting to stay up to date about challenges encountered by other parts of the networking industry, at scales way different than the ones we are used to.

We learned for example what CXL is about, what it brings and the effort that are made to design new networking hardware around this technology to change the way we think about datacenter networking.

When we attended Netdev 0x13 in 2019, QUIC was one of the hot topics. This year, Homa was under the spotlight with talks on what it is, and how this new protocol could address some of TCP’s problems.

Like all previous editions, we learned all the progress that were made with TC and its future, new ways of bypassing the kernel stack, BPF integration in the kernel, along with XDP which continued to be more and more powerful.

Another hot topic in the kernel is the introduction of the Rust language, and the network subsystem is a pretty relevant target for the new features brought by the language. As a consequence, Rust subsystem maintainers Miguel Ojeda and Wedson Almeida Filho gave an overview of Rust benefits compared to traditional C code, and then showed a step-by-step implementation of a kernel-side TCP server module. While this example is not perfectly representative of classic network-related drivers we usually write, it was a nice showcase of current state of kernel APIs abstractions in Rust.

We also discovered the new use-case that is now driving most of the datacenter networking efforts, which is without surprise AI and Machine Learning. Turns out, if you want your ChatGPT to answer up-to-date replies without having to wait for too long, you need a powerful and well-organized datacenter for the training part, and networking engineering takes a big part in it to keep all those GPUs fed at a relevant pace.

This lead to the devmem TCP effort, which started to feel a bit familiar for us as it uses dma-buf, which we also sometimes use on multimedia pipelines. The ML and AI topic was introduced to us by the wonderful Keynote session given by Manya Ghobadi, who got all the audience captivated by how AI and ML works, what AI workloads requires in terms of network traffic scheduling, datacenter topology and computing hardware that uses optical computing.

On the final day, we even had a visit from Jakub Kicinski (one of the co-maintainers of Linux networking tree), presenting what he had been working on, and gave us an update on the netdev development statistics (and basically, his main point is that we do need to review more patches).

For the first time, there was a talk from Bootlin at netdev, as Maxime presented one of the topics he’s been working on lately : Improving multi-PHY and multi-port interfaces support. Although it was one of the only talks focusing on the low-levels aspects of the Ethernet stack, it triggered some discussions and interest from the community, which will help further improving the ongoing work.

The slides and videos of the event will be published at some point in the future, we will for sure mention this to our readers when it becomes available.

We’ll conclude this short feedback by thanking once again the Netdev Board members, organizers, speakers and the audience for this great event.

We’ll come back 🙂

Linux 6.6 released, Bootlin contributions

Linux 6.6 was released yesterday, so this is the time for our usual blog post about our contributions to this release. Before that, to get an overall idea of what went into Linux 6.6, we recommend reading the articles from LWN.net covering the Linux 6.6 merge window: part 1 and part 2. The KernelNewbies page is perhaps a little bit less rich than it used to be, but still relevant.

On our side, this time around we contributed 68 changes to this release:

  • Alexandre Belloni, as the RTC subsystem maintainer, submitted a few asorted patches touching various drivers in this subsystem
  • Alexis LothorĂ© pushed some patches extending the rzn1-a5psw Ethernet switch driver with VLAN support and port_bridge_flags support. These patches were initially written by ClĂ©ment LĂ©ger but had not been accepted until now.
  • HervĂ© Codina got his audio-iio-aux driver merged, which allows the ASoC subsystem (for audio devices) to use IIO devices, such as a potentiometer. This came together with a number of fixes/improvements in the IIO subsystem. HervĂ© also fixed some reference counting issues in several I2C mux drivers.
  • Miquèl Raynal pushed to the finish line a patch written several years ago by Bootlin engineer Kamel Bouhara, who hadn’t been accepted until now. This patch adds a sysfs interface that allows to retrieve the reset reason on Microchip ARM platforms
  • Luca Ceresoli fixed some issues in two DRM panel drivers and also fixed a regression in the NVidia Tegra camera interface driver
  • Miquèl Raynal did a number of different, unrelated, contributions:
    • support for the EDT ET028013DMA display panel to the existing sitronix-st7789v driver, which required quite a few preparation changes
    • fix a clock polarity issue in the DRM driver for the display controller used in Microchip ARM platforms
    • improve many small aspects of the qcom NAND controller driver
    • improve the handling of nvmem layouts in the nvmem subsystem
    • fix an issue in the SJA1000 CAN controller driver that would cause the HW to stall after an overrun on some platforms
  • Paul Kocialkowski contributed a few small asorted fixes in the media subsystem documentation

Here are the complete details of our contributions:

Yocto Project Summit 2023.11: 2 Bootlin talks

The Yocto Project regularly organizes an-online conference called the Yocto Project Summit. The next edition, Yocto Project Summit 2023.11 will take place on November 28-30, from 12:00 to 18:00 UTC, and at just $40, attending is really affordable.

Yocto Project Summit

Bootlin is not only a big user of the Yocto Project, but also a significant contributor to the project, so we’re happy to announce that our two talk proposals for the Yocto Project Summit 2023.11 have been accepted. Bootlin engineers will therefore deliver the following talks:

If you are a user of the Yocto Project, or intend to become one, we can only recommend you to attend this event. And of course, if you need training on Yocto Project, or engineering/support services, do not hesitate to contact us!

Internships at Bootlin in 2024

Bootlin is happy to announce its list of internship topics for 2024, which are open to students in engineering schools or similar, from France or the European Union. Our internship booklet is in French as most of our interns come from France.

The proposed topics are as follows:

  • Drivers and hardware support in Linux or U-Boot
  • Improvement of the Elixir code navigation service
  • Security tracking of Linux BSP
  • Implementation of a reference secure embedded Linux OS
  • Addition of USB composite gadget support in U-Boot
  • Improvement of IEEE 802.15.4 support in Linux
  • Open topic related to embedded Linux

Bootlin is a company with a strong open-source focus: the results of the internships will be contributed to the corresponding open-source projects and will lead to the publication of blog articles or public presentations. In addition, interns will have the opportunity to work with a team of engineers with strong expertise in embedded Linux and open-source contributions.

The internships are offered for 2024, with flexible start dates, but for a minimum duration of 4 months. These internships will take place either in our offices in the Toulouse area (Colomiers) or in the Lyon area (Oullins), in order to be integrated into our team of engineers. These internships are open to all students who are citizens of the European Union.

Although several internship positions are offered, Bootlin will host a maximum of 2 interns simultaneously, in order to provide interns with high-quality supervision and guidance from Bootlin’s engineers.

For any questions or applications regarding these internship offers, please contact jobs@bootlin.com.

Bootlin at Netdev 0x17, THE Technical Conference on Linux Networking

VancouverBootlin will be at the Netdev 0x17 conference, subtitled THE Technical Conference on Linux Networking. It is indeed one of the major event for developers working on the networking side of the Linux kernel to gather and discuss current and future topics. This year, the conference will take place from Oct 30 to Nov 3 in Vancouver, Canada.

Bootlin is involved in a number of Linux kernel networking developments: development and/or improvement of Linux kernel drivers for Ethernet MACs, Ethernet PHYs, WiFi chips, support for SFP, for Ethernet switches, for PTP offloading, for MACsec offloading, improvements to the 802.15.4 stack, and more. As such, it is very relevant for us to meet the Linux kernel networking community, present our work, and understand where things are heading to in the networking stack.

Our engineers Maxime Chevallier and Alexis Lothoré will both attend the conference. In addition, Maxime will be presenting a talk titled Improving multi-phy and multi-port interfaces:

This talk will describe current use-cases where one MAC is connected to multiple PHYs (chained, or in parallel) and multiple front-facing ports, either through multiple PHYs or through a single multi-port PHY. There exist support for some of these scenarios already, but it is limited by the fact that the PHY device is hidden behind a net_device from userspace’s point of view. We therefore can’t configure an individual PHY when multiple PHYs are present on a link (through SFP transceivers for example), and selecting which front-facing port to use is also limited. This talk will describe ongoing work to support these complex topologies, the challenges faced and expected improvements.

We look forward to attending this event in a few weeks time!

Systemd, read-only rootfs and overlay file system over /etc

Systemd is a popular init system, used to bootstrap user space and manage user processes. It now replaces several Linux utilities with its own components like log management, networking, time management, etc. There is even a bootloader component now. Systemd is obviously ubiquitous nowadays for desktop/server Linux distributions, and is also commonly used on embedded devices to benefit from features such as parallel startup of services, monitoring of services, and more.

In a recent project that uses Buildroot as its build system, we have used systemd with the storage consisting of a read-only root filesystem (SquashFS) and an overlay file system (OverlayFS) mounted on /etc. While doing this, we faced two issues with the use of OverlayFS on /etc:

  • /etc/machine-id file management. This file is created during the first boot by systemd, and if the root filesystem is read-only, it will bind mount it to /run and wait to have read-write access to create it (see more details). In that case, the machine-id file is re-generated at each boot (because /run is a tmpfs, which means that the machine identification changes at each boot, which is not necessarily desirable. On the other hand, we don’t want to machine-id file to be part of the SquashFS filesystem because the SquashFS filesystem is identical on all devices, while the /etc/machine-id file is unique per device. So ideally, we would like this machine-id file to be stored in our OverlayFS, generated during the first boot. The issue is that reading the machine-id file is done very early by systemd, before we get the chance to mount the OverlayFS.
  • We wanted to be able to add or modify systemd services using the OverlayFS. Systemd parses the service files at early init and executes them according to their order and dependencies. The service mounting the filesystems from /etc/fstab and any other services is started after such parsing, which is too late. We could think of running daemon-reload from a custom service once mounting was complete, but this is not really a stable solution, as
    Lennart Poettering commanted on in a short e-mail thread about this issue.

The solution suggested by Lennart, and elsewhere on the wider Internet is to mount the OverlayFS from an initramfs, which allows to have it setup before systemd even starts. As we use Buildroot and using an initramfs adds complexity by requiring a separate configuration to manage multiple images. This was overkill in our case, just for setting up the overlay. The solution we eventually chose was to create an init_overlay.sh script which is started as init before systemd, by adding init=/sbin/init_overlay.sh to the kernel command line:

#!/bin/sh
mount -t proc -o nosuid,nodev,noexec none /proc
mount -t sysfs -o nosuid,nodev,noexec none /sys
mount /dev/mmcblk0p2 /mnt/data
mount -t overlay overlay -o lowerdir=/etc,upperdir=/mnt/data/etc,workdir=/mnt/data/.etc-work /etc
exec /sbin/init

Hopefully, this will be useful to others. Of course, we’re also curious to hear if others faced the same issue, and discover how they solved this. Let us know in the comments.