Back from the Embededded Linux Conference: selection of talks #1

As we wrote in a previous blog post, 11 engineers from Bootlin attended the Embedded Linux Conference in Seattle in April. We have a tradition after such an event to share with you a selection of talks that we have found useful. In order to achieve this, we ask each of our engineers who participated to the conference to pick one talk they would like to highlight, and write a short summary/feedback about the talk. In this first installment of this series of blog posts, we’ll share our selection of 4 first talks.

Talk Real-Time Linux: What Is Next?, by Daniel Bristot de Oliveira, Sebastian Siewior, Kate Stewart

Talk chosen by Bootlin engineer Maxime Chevallier, who is also the author and trainer of our Real-time Linux with PREEMPT_RT training course.

As with every release, the Preempt-RT patchset is closer and closer to being fully merged in to the mainline tree. While there are still some issues left to be addressed, namely printk, the panel discussion was a good opportunity to discuss what’s next. What can we expect when Preempt-RT is merged, and what are the next steps going to be ?

Over the years, Preempt-RT and its developer community have used this cozy corner of the Linux kernel to work on new, advanced and complex features that eventually made it into the mainline kernel. Things aren’t going to change much on that regard, efforts to improve scheduling, or lately, RT-throttling, are now being made in the mainline tree.

Another point that was brought-up is the continuing need to educate users on how to use Preempt-RT. For people who need a Real-Time linux-based operating system, using Preempt-RT is just the beginning of the journey, most of the work really is about knowing how the entire system behaves in real-life conditions, and avoid latencies coming from the hardware, the kernel, or user applications. This won’t change much after Preempt-RT lands mainline, the need for education stays the same. Some propositions made to improve on the realtime linux wiki were made, to use it as a centralized place to share configuration examples and best practices.

Tools such as rtla have however made it much easier for users to gather and share information on the issues they encounter, this tool can be expected to gain more and more traction, and in turn help users get familiar with Real-Time debugging with more ease.

We’re all very excited to see the hard work of the Preempt-RT developers make it into mainline Linux, but it was clear that this is going to be just another step in the long journey of the Real-Time Linux project.

Compound Interest – Dealing with Two Decades of Technical Debt in Embedded Linux, by Bartosz Golaszewski, Linaro

Talk chosen by Bootlin engineer Luca Ceresoli

In this talk with some humor and lots of serious considerations, Bartosz discussed the high cost of “technical debt”, i.e. dealing with the cost of suboptimal solutions used in software development and the accumulation of workarounds and hacks over time to circumvent the effort of fixing the initial design.

By implementing a suboptimal software component a developer borrows time, which will have to be paid back when using this suboptimal software component (this is “technical debt”, compared to “financial debt”). Later on one can add workarounds and hacks instead of fixing the original design flaw, thus accumulating “compound interest”. The problem is much worse when the suboptimal design touches programming interfaces (especially the user space APIs!).

The problem with APIs is that after they are added people will inevitably start using them in new ways, some which were not even considered initially. This makes future restructuring more and more complex as none of the use cases should be broken.

In the Linux kernel this usually happens when a subsystem has a lot of users, so modifying its internal API is extremely costly. Another common reason is that people implementing a new subsystem tend to start by copying an existing subsystem, thus copying the same errors – and duplicating technical debt. Bartosz finds the lack of a good deprecation mechanism in the Linux kernel exacerbates this problem.

The GPIO subsystem, which Bartosz maintains, is a good example. It has a long history, predating device tree and the device driver model, it has lots of providers and many more users, finally it started as a very naïve set of function prototypes and evolved over time. It provides a number-based sysfs interface that is very problematic, on top of which a new, better interface has been developed, called gpiod.

The large variety of GPIO users (some in atomic context) and GPIO providers (some which may sleep) created a long-standing issue as serialization of calls to the GPIO subsystem appeared close to unfeasible. However Bartosz found a good solution using SRCU, which has been merged in v6.9. One part of the technical debt has been paid back, finally.

The debt that is the hardest to pay back is the one originating from user space APIs, which just cannot be removed for backward compatibility. People have been long encouraged to use the new gpiod interface but some still complain that it has no easy way to set a GPIO output persistently. Bartosz is working on a solution to this by implementing a D-Bus protocol to set GPIOs and a related daemon that is able to persistently keep GPIOs set. Bartosz will be very happy if people test it and report their findings! Once finished, this will leave no excuses to switch to the gpiod user API.

Overall we found this talk enlightening about how bad programming design creates technical debt, the difficulty to solve it and what to do to minimize it in the first place.

How (Not) to Get the Vendor Driver Merged Upstream, by Dmitry Baryshkov, Linaro

Talk chosen by Bootlin engineer Bastien Curutchet

In this talk Dmitry Baryshkov presented the fundamentals of a good contribution to the Linux kernel.

He started with the classic steps to fulfill when you want to contribute and mention the checks that have to be made before submitting a patch as:

  • ./script/ to check code style compliance
  • make dtbs_check and make dt_bindings_check when changes are made in Device Tree bindings
  • make W=1 to fix all warnings

Then, Dmitry presented the main b4 commands that can be used to prepare and send your patches upstream. He also gave advice about the patch series contents: kernel must compile and work after application of each patch; it is better to split a big patch into small ones and a big patch series into smaller ones; commit log must describe why the introduced change is needed. Actually, Krzysztof Kozlowski also insisted on this last point in another good talk: giving context to maintainers help them during their review.

Finally he talked about the good and bad behavior habits when contributing. His first advice is ‘Don’t fear asking questions‘, if there is something that you don’t understand in a reviewer’s comment or question: don’t ignore it and ask for details.
The second thing to do is to leave people time to review. It is not needed to send a new iteration right after the first comment you get. It’s better to leave others time to add more comments (or even debate about some points) before sending your next iteration. The golden rule would be: never do more than one iteration per day, even one per week for a big patchset. According to him, if you don’t get any answer at all during two weeks, it’s ok to resend your patch series or to ping the maintainer.

As a conclusion, Dmitry pointed out that the first contribution is the hardest, and things will get easier and easier with following contributions.

As a fairly new contributor to the Linux kernel, I really appreciated this talk because Dmitry went through every question I asked myself and my colleagues before and during my first contribution so I’d say this talk can be really useful for someone who would like to start contributing to the Linux kernel project.

Talk Maximizing SD Card Life, Performance, and Monitoring with KrillKounter, by Andrew Murray

Talk chosen by Bootlin engineer João Marcos Costa

The apparent unreliability of SD Cards is a usual complaint from customers when they find that data has been corrupted, or that the throughput announced by the manufacturer is far from reality.

In this talk, Andrew Murray starts with an overview of what a typical SD Card contains: the card’s connectors, a microcontroller (the SD Controller), and a NAND memory chip. However, as he points out, NAND is fundamentally unreliable.

Such unreliability comes mainly from the degradation of the semiconductor that composes the Floating-gate MOSFET. This transistor is the base for both NAND and NOR Flash, each one with inherently different levels of granularity for writing operations. The talk focuses on NAND Flash, as it can be found everywhere: SD Cards, eMMC, SSD, USB sticks, etc.

NAND’s access limitations are a consequence of its write granularity: a whole block of pages need to be erased so you can write into a single page. The presentation illustrates with an example where a 3×3 block (i.e., with 9 pages) needs to write into its first page. A first approach would be writing an entirely new block: the “new” first page would be written into this block, and the previous eight pages would be copied from the old block. The drawback here is called write amplification: we ended up with 9 write operations instead of only one.

The lifetime of NAND Flash is measured in Program/Erase cycles (P/E cycle), and this value changes according to the memory cell architecture: SLC (Single-level cell, 1 bit per cell) has an approximate lifespan of 10000 P/E cycles, MLC (Multi-level cell, 2 bits per cell) and TLC (Triple-level cell, 3 bits per cell) have approximative lifespans of 3000 P/E and 1000 P/E respectively.

One of the talk’s main points is the experimental testing exploring the lifetime of 4 regular 8Gb MLC consumer SD Cards and writing to them until they fail. Both sequential and random writes were used. As for the sequential writes, the block size used was 512Kb. As for the random writes, three different values were used: 4Kb, 128Kb and 512Kb. The overall conclusions were that sequential writes have a fairly higher throughput and lower degradation. For the random writes, the larger the block size, the higher the throughput and the lower the degradation levels. Overall, the conclusion is that sequential large accesses are usually better. This whole experiment was tracked with the Open Source daemon KrillKounter, responsible for logging block layer statistics and allowing to determine the wear on a SD Card.

The talk was particularly instructive, as it starts from a practical overview of SD Cards and NAND Flash, then dives for a moment in electronics specifics to explain how the oxide degradation works, and finally presents us with an experiment to confront theoretical values of the SD Cards lifespan. As a bonus, it provides us with KrillKounter, a tool to analyze the wear on SD Cards.

Linux 6.9 released, Bootlin contributions inside!

Linux 6.9 was released last Sunday, and as usual we refer our readers to the excellent coverage of the Linux 6.9 merge window (part 1 and part 2) to get a good overall picture of the improvements and new features brought by this release.

On our side, we contributed a total of 119 commits authored by Bootlin engineers, but we also merged a total of 95 patches from other contributors, as several Bootlin engineers as also maintainers of various drivers/subsystems in the Linux kernel.

Here is the highlight of our contributions:

  • Alexandre Belloni, as the RTC subsystem maintainer, submitted a number of improvements to various RTC drivers, with a particular focus on the rtc-ds1511 driver
  • Alexis Lothoré continued his work on the wilc1000 Wireless driver, for the Microchip ATWILC1000 chip. Alexis is going through a significant backlog of out-of-tree patches, to bring the mainline driver on par with the vendor driver
  • Grégory Clement contributed the initial support for the Mobileye EyeQ5 processor, used in autonomous driving applications
  • Hervé Codina, after submitting in Linux 6.8 a new WAN framer subsystem in Linux 6.8, he contributed drivers/net/wan/fsl_qmc_hdlc.c a new WAN driver to perform HDLC using the PowerPC QMC hardware block, as well as a sound/soc/codecs/framer-codec.c, which is an ASoC codec driver that works on top of the WAN framer subsystem
  • Hervé also contributed some fixes to the devlink mechanism
  • Jérémie Dautheribes contributed some improvements to the TI drivers/net/phy/dp83822.c Ethernet PHY driver
  • Köry Maincent changed the PTP subsystem to use the XArray data structure instead of simple IDA, and contributed a fix to the Linux kernel regulator subsystem, in preparation for his work on PoE
  • Luca Ceresoli contributed new trace events in the ASoC subsystem, fixed a bug in the I2S driver for Rockchip platforms, and fixed some description text in a Device Tree Binding
  • Louis Chauvet fixed an issue in the Xilinx XDMA dmaengine driver
  • Maxime Chevallier contributed documentation fixes in the phylink subsystem
  • Michael Opdenacker marked ext2 as a deprecated driver
  • Miquèl Raynal contributed some fixes to his implementation of continuous read support for parallel NAND, and also some fixes to the same Xilinx XDMA dmaengine driver as Louis Chauvet
  • Richard Genoud arrived at Bootlin, and updated his mailmap entry
  • Thomas Richard continued his work on suspend/resume support on TI K3 platforms
  • Last but not least Théo Lebrun is for this release our biggest contributor, with 46 patches. All his work is centered around enabling support for Mobileye EyeQ processors, with a focusing on pinctrl, I2C, SPI and GPIO. The SPI controller used is from Cadence, the I2C and GPIO blocks are the same as the ones available on Nomadik SoCs, so Théo mostly improved/extended those drivers to support the Mobileye EyeQ case.

Here is the complete details of our contributions:

Bootlin engineer Louis Chauvet at Linux Display hackfest

2024 Display Next HackfestFrom May 14 to May 16, Igalia is organizing the 2024 Display Next Hackfest, an event where talented developers will gather to explore the latest technologies and trends in the Linux Display Stack.

As explained on the event website:

It has an unconference format where participants propose topics for presenting, roadmapping, discussing and examining together. It aims to unblock bottlenecks, design solutions, raise pitfalls and accommodate the needs of each layer of the display stack. Participants should feel free to propose any topic which interests them. Some topics from the previous edition include: HDR and color management, frame timing and variable refresh rate (VRR), atomic flips, testing and CI, etc.

Bootlin engineer Louis Chauvet, who has started contributing to the Linux kernel VKMS driver, and is starting to work on IGT and the latest version of the Chamelium CI testing hardware, will participate to this hackfest, together with many developers from Igalia, Redhat, Intel, Google, RaspberryPi, AMD, ARM, Collabora and more. This will allow us to discuss current developments and topics, and meet the relevant developers of the Linux graphics/display community.

Yocto 5.0 Scarthgap released, Bootlin contributions inside

Yocto Project SummitThe latest release of the Yocto Project, version 5.0, code named Scarthgap has been published a few days ago. The release notes provide the best summary of what’s new in this release. Being a Long Term Support (LTS) release, it will be maintained during 4 years with bug fixes and security updates, which makes this release particularly important for a large number of embedded Linux projects and products.

At Bootlin, we are using Yocto for a large fraction of the Linux Board Support Packages that we develop, maintain and upgrade for our customers. But we’re not only users of Yocto: we’re also contributors and maintainers. In this blog post, we’ll highlight our contributions to this release, which take various forms.

Continue reading “Yocto 5.0 Scarthgap released, Bootlin contributions inside”

Our talks at Embedded Open Source Summit 2024

The Embedded Open Source Summit 2024 took place on Apr 16-18 in Seattle, with many talks on a wide range of embedded Linux topics. 11 engineers from Bootlin participated to this conference and four of us gave talks, for which we are happy to publish the slides and videos in this blog post.

Bootlin team at Embedded Open Source Summit 2024
Bootlin team at Embedded Open Source Summit 2024

Continue reading “Our talks at Embedded Open Source Summit 2024”

Getting started with Zephyr

Zephyr is an open-source real-time operating system, used mainly in embedded devices, with a focus on small systems, thanks to its very small footprint.

This post is a quick startup guide to show how to run Zephyr on two different boards, from two different vendors:

In this post, we will show how to install all the tools needed to build and run Zephyr, then run some samples, until we get access to the Zephyr shell over USB.

Continue reading “Getting started with Zephyr”

Welcome to Richard Genoud!

Welcome on board!We are very happy to welcome in our team Richard Genoud, who joined us at the beginning of April, and already participated to his first conference with Bootlin by attending the Embedded Linux Conference this week.

Richard started using Linux in 1998, and his journey working with embedded Linux started in 2001, so Richard brings to our team over 20 years of experience working on embedded Linux systems. In his previous positions, Richard has worked on all aspects of embedded Linux, from porting and improving bootloaders, porting the Linux kernel to various hardware platforms, using build systems such as Buildroot, developing user-space applications. Richard helped ship and maintain products in the field for many years.

Richard is also an open-source contributor: he contributed support for the NAND ECC controller on Microchip platforms, contributed improvements to the UART driver for Microchip platforms, for which he is the official Linux kernel maintainer, but Richard also made contributions to U-Boot, dnsmasq, msmtp, Buildroot, connman, Busybox or mtd-utils.

For more details, see Richard’s profile on our site.

Linux 6.8 released, Bootlin contributions

The Linux 6.8 kernel has been released on March 10 by Linus Torvalds. As usual, we definitely recommend the coverage by LWN of the merge window for this release cycle to get a good grasp on the most important new features: first half and second half. Some work from Bootlin is briefly mentioned in those articles, such as the support for Lantiq PEF2256 (FALC56) framers, Lantiq PEF2256 (FALC56) pin controllers, and Techwell TW9900 video decoders.

With a total of 135 commits contributed by Bootlin engineers during this release cycle, we have been much more active than for the previous 6.7 release. This allows Bootlin to show in the recently published Development statistics for 6.8, as the 17th contributing company by number of commits, and 13th contributing company by number of changed lines.

Continue reading “Linux 6.8 released, Bootlin contributions”

ext2 filesystem driver now marked as deprecated

31 years after the start of its career in 1993, it’s time for ext2 to retire. Here we are talking about the driver for this filesystem, not exactly the filesystem itself. Continue reading to understand the subtle difference.

It’s the ext2 filesystem driver that will be marked as deprecated in the upcoming 6.9 Linux kernel. The main issue is that even if the filesystem is created with 256 byte inodes (mkfs.ext2 -I 256), the filesystem driver will stick to 32 bit dates. Because of this, the driver does not support inode timestamps beyond 03:14:07 UTC on 19 January 2038.

Continue reading “ext2 filesystem driver now marked as deprecated”

Updated Buildroot support for STM32MP1 platforms, ST BSP v5.0

We continue the support of the buildroot-external-st project as presented in our first announce, which is an extension of the Buildroot build system with ready-to-use configurations for the STMicroelectronics STM32MP1 platforms. We already published several updates of this support, with the LTS 2022.02 and version 4.0 of ST BSP version update and the LTS 2022.02.7 and version 4.1 of ST BSP version update.

More specifically, this project is a BR2_EXTERNAL repository for Buildroot, with a number of defconfigs that allow to quickly build embedded Linux systems for the STM32MP1 Discovery Kit platforms. It’s a great way to get started with Buildroot on those platforms.

Today, we are happy to announce an updated version of this project, published under the branch st/2023.02.10 at This new version brings a number of additional features:

Continue reading “Updated Buildroot support for STM32MP1 platforms, ST BSP v5.0”