Thomas Petazzoni is Bootlin's co-owner and CEO. Thomas joined Bootlin in 2008 as a kernel and embedded Linux engineer, became CTO in 2013, and co-owner/CEO in 2021. More details...
We are pleased to welcome two additional engineers to our team based in Toulouse, France: Romain Gantois and Louis Chauvet.
Romain Gantois graduated from ISEP and completed his final internship at Bootlin during which he developed and published Snagboot, the generic and open-source board recovery and reflashing tool, and worked on an upstream Linux kernel driver for a Qualcomm Ethernet switch (patches will be submitted soon!). Following this internship, Romain is joining our team as a full-time embedded Linux and Linux engineer.
Louis Chauvet graduated from INSA Toulouse. He completed his final internship abroad, during which he worked on developing in Rust, in particular the development of Linux kernel drivers in Rust. Louis is also joining us as a full-time embedded Linux and Linux kernel engineer.
Both Romain and Louis are experienced Linux users and developers, with a solid education in low-level and embedded systems development. They will help us address more embedded Linux projects from our customers on a wide variety of topics, and are already benefiting from our training courses and the interaction with our senior engineers to quickly gain even more knowledge and experience.
As we reported in a previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post: first post, second post, this one being the third of the series.
rtla timerlat: Debugging Real-time Linux Scheduling Latency
Talk by Daniel Bristot de Oliveira, chosen by Bootlin engineer Maxime Chevallier.
Talks related to real-time linux debugging are pretty common at ELCE, I gave one myself in 2017 and I’ve been attending most of them since then. Besides a headache, what I could get from attending all these talks is that this topic is complex, time consuming, and that there’s a lot of different methodologies one can use to find the cause of these elusive problems.
Users who aren’t very familiar with the inner workings of the Linux Kernel can ask for help on mailing-lists, and the reply usually asks for a trace. This is where things get complicated, the Linux kernel tracer is very powerful, but can drown users in a flood of trace events from which it is difficult to extract the relevant data.
Hopefully, Daniel’s talk is going to make this kind of talk less common, as the tool he wrote and presented, rtla, makes it easy to gather important information about the cause of undesired latencies. By using cleverly placed trace-points, in-kernel testing tools (timerlat and osnoise) and an automated trace analyzer, rtla can not only detect latencies as cyclictest would, it can also give you what caused the latency. If it’s a blocking problem, rtla tells you which process is blocking your task. If it’s an interference, rtla will tell you which task or interrupt caused the latency, and can even detect if the hardware itself is the culprit.
For developers, this tool is also a perfect way to gather user feedback and bug reports that are small, precise and easily reproducible.
I therefore strongly recommend checking out Daniel’s talk and his dedicated blog article.
Zbus – the Lightweight and Flexible Zephyr Message Bus
Talk by Robrigo Peixoto, chosen by Bootlin engineer Thomas Perrot
Zbus is a new message bus for Zephyr allowing threads to communicate to many others, easily. This bus allows to implement several bus topologies:
one-to-one
one-to-many
Many-to-many
In addition, it can be used on very constrained systems.
In this talk, Rodrigo explained in detail how Zbus works, through a few examples. A thread can read or publish in bus channels, and when a message is published into a channel:
The Listener’s callbacks are executed
A notification is put to the subscriber’s queues
Then the subscriber will be executed by priority order
The bus is managed by a dispatcher, named Virtual Distributed Event Dispatcher (VDED) that is robust to priority inversion.
We found Zbus to be a very interesting feature because before there was no easy way to implement one-to-many and many-to-many topologies, but also one-to-one communications without having to manage the problems of inverting priorities and to use FIFO, LIFO, pipe, etc.
Linux Power ! (from the Perspective of a PMIC Vendor)
Talk by Matti Vaittinen, chosen by Bootlin engineer Kamel Bouhara.
PMICs (Power Management Integrated Circuit) are a key component of low power embedded systems as they often handle complexity in controlling various power voltages required by SoCs. In his talk Matti Vaittinen started by depicting the various devices that can be embedded in a PMIC (Power Management Integrated Circuit): watchdog, RTC, GPIOs are examples of such extra functionalities. He reminded us the reason why such devices are best fitted in the Linux MFD subsystem to take advantage of existing code. However the main subsystem used to implement support for a PMIC is the regulator subsystem and the talk gives us a good understanding of how it works, the concept of provider/consumer, how to register multiple regulators for a PMIC and how to handle specific events. A focus is made on error detection and how over current errors are reported over three categories:
PROTECTION : hardware level errors reported when protection limit is reached
ERROR: Unrecoverable errors that don’t directly involve hardware shutdown.
WARNING: System is still recoverable but requires specific action to be taken
Some PMICs also provide IRQs to notify errors or events and the kernel provides a helper function to handle such notifications and map them to specific actions depending on their severity.
Overall, we found this talk interesting to understand bettert the features provided by PMICs, and how these features are supported by Linux.
Linux 6.5 was released yesterday, with as usual over 10,000 commits from a large number of contributors. We recommend reading LWN.net articles on the merge window (part 1, part 2), but also the CNX Software page that focuses on embedded-related improvements.
Bootlin contributed 76 commits to this kernel release, putting us as the #26 contributing company. This time around, our main contributions have been:
The large stack of patches from Luca Ceresoli on the NVidia Tegra camera interface driver finally landed: they add support for the Tegra20 parallel camera interface to the existing driver, which required a lot of changes to the driver that was so far only support Tegra210 CSI. This work allows one of our customers, who was stuck on an old vendor NVidia kernel to an upstream Linux kernel.
Hervé Codina contributed a driver for the Renesas X9250potentiometer, in the IIO subsystem. This will be followed in Linux 6.6 by a glue driver that allows to expose an IIO device as an auxiliary device in the ALSA subsystem, allowing this potentiometer to be used in audio applications
Alexis Lothoré contributed support for the Marvell MV88E6361 Ethernet switch into the existing mv88e6xxx DSA driver
Maxime Chevallier contributed a new regmap-based MDIO driver, which required some changes in the regmap code. This allows the Altera TSE driver to use the existing Lynx PCS driver, and drop the custom Altera TSE PCS driver. Finally, the stmmac Ethernet driver is modified to be able to use the Lynx PCS driver as well. Quite an adventure to finally get proper PCS support with stmmac
Miquèl Raynal contributed improvments in the 802.15.4 stack, especially related to scanning support.
Miquèl Raynal contributed fixes to the sja1000 CAN driver (to avoid overrun stalls on Renesas processors), to the SPI subsystem (to avoid false timeouts for long transfers), to the DMA engine driver for Xilinx XDMA IP, and a few more.
Miquèl Raynal also continued his effort of improving the Device Tree bindings for MTD NAND controllers
Luca Ceresoli added sound card support to the MSC SM2-MB-EP1 carrier board, which runs a i.MX8MP SoM, and he also fixed the timings for one of the panels supported by the simple-panel driver
Here are the details of all our changes that went into Linux 6.5:
The Embedded Recipes conference was started a few years ago, modeled after the popular Kernel Recipes conference. Both events follow an unusual but very interesting format: a single-track conference, with a carefully chosen set of talks/speakers, and a limited number of attendees. These “design choices” give those events a different atmosphere than larger events, whether they are corporate-driven or community-driven.
This year, the Embedded Recipes conference will take place in Paris on September 28-29, with two full days of talks, but also opportunities to connect which are facilitated by the smaller audience and single-track nature of the event. The schedule is now online.
Bootlin has decided to support this year’s edition by being one of the Chef sponsors, contributing financially to the sustainability of this conference. In addition, Bootlin engineer Romain Gantois will give a lightning talk about Snagboot, our recently released open-source and HW agnostic tool to recover and reflash embedded platforms.
If you’re interested in meeting with Bootlin folks, do not hesitate to reach out. Remember: we are still hiring, and looking for engineers with embedded Linux and/or Linux kernel experience to join our team. Meeting at this conference would be a great opportunity to start the conversation!
In the mean time, we can only recommend one thing: plan on attending the conference, and register now!
End of June, we announced the availability of a brand new training course, Embedded Linux Audio, which is targeted at engineers working with audio on embedded Linux systems, and that covers topics ranging from audio hardware, audio support in the Linux kernel (ASoC, DAI and codec drivers, Device Tree representation), audio support in user-space (alsa-lib, alsa-utils, PipeWire, GStreamer).
We are pleased to announce today that the training materials are now available for download. This again shows our commitment to sharing all our training materials for free, under an open-source license (Creative Commons BY-SA).
This training course has already been given earlier this year to a private customer, and we are releasing the materials before the very first public session, which will take place on September 11 to September 14, and which will be taught by Bootlin engineer and COO Alexandre Belloni. If you’re interested, registration is open, we have a few seats remaining.
Earlier this year, the BeagleBoard.org foundation announced the availability of a new development board, the BeaglePlay, centered around the Texas Instruments AM625 ARM64 processor. Bootlin has for a long time supported the BeagleBone Black as one of the hardware platforms used in the practical labs of its training courses, and we are happy to announce today that we now have added support for the BeaglePlay to our most popular course, the Embedded Linux system development course.
This means that participants to our Embedded Linux system development course can now choose to perform the practical labs on the BeaglePlay, experiencing a new and modern hardware platform, based on an ARM64 processor. This applies not only to participants choosing to purchase our course, but also to everyone else in the world, as our training materials are all freely available and distributed under an open-source license. The materials are all available on our Embedded Linux system development training course page:
We would like to thank Clément Ramirez who has worked with us on this effort of porting this course to the BeaglePlay platform.
“Bootlin is a leading provider of high-quality, in-depth Linux education. What amazes me is how much they not only support open source with outstanding teaching, but how much they stand behind the principles of open source by sharing their teaching materials and contributing to the open source projects that drive them.” said Jason Kridner, founder of BeagleBoard.org. “BeagleBoard.org shares these principles and that is why I believe BeaglePlay is an excellent choice. BeaglePlay doesn’t just run open source software, the hardware itself is open source with detailed documentation on the board and associated Texas Instruments AM625 system-on-chip, with availability to enable hobbyists and professionals alike from prototype to production. I couldn’t be more excited.”
Bootlin’s Embedded Linux course, like all our training courses, is available in 3 different options:
Public on-line sessions, delivered by video-conference, with a per-participant registration, and sessions organized at dates scheduled by Bootlin. See this page for more details, dates and registration process
Private on-line sessions, also delivered by video-conference, organized privately for your team, at the date of your choice. See this page for more details
Private on-site sessions, where one of our experienced engineers and trainers travels to your location, to teach your team in person. See this page for more details
In both our private on-line sessions and private on-site sessions, our customers are free to chose, among our set of supported platforms, the HW platform they would like to use for the practical labs, and this now includes the BeaglePlay. In our public on-line sessions, the trainer demonstrates the practical labs on one particular platform, but participants are able to reproduce the labs on the platform of their choice among our supported platforms.
With this new development, we look forward to continue our mission of helping the broader engineering community get trained on embedded Linux technology and expand the number of users and contributors to open-source technologies and communities.
As we reported in previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post: first post, this one being the second post.
We are happy to announce that we have just published a new update of our freely available toolchains at toolchains.bootlin.com, version 2023.08.
For the record, we provide pre-built cross-compilation toolchains that work on x86-64 Linux machines, and targeting 43 different CPU architecture variants, with support for all 3 major C libraries: glibc, musl and uClibc-ng. For each toolchain, we provide two versions: a stable one that uses GCC/binutils/GDB versions next to the last, and a bleeding-edge one that uses the very latest GCC/binutils/GDB versions.
In this 2023.08 release, we have:
Updated the bleeding-edge toolchains to gcc 13.2, binutils 2.41, gdb 13.2, kernel headers 5.10, glibc 2.37, musl 1.2.4 or uclibc-ng 1.0.43
Updated the stable toolchains to gcc 12.3, binutils 2.40, gdb 12.1, kernel headers 4.14, glibc 2.37, musl 1.2.4 or uclibc-ng 1.0.43
Marked the sparcv8 toolchain as obsolete as sparc support in GCC has been broken for several releases, and the last working version of GCC for sparc has been dropped from Buildroot
A special thanks to Romain Naour from Smile who helped investigate and resolve some of the issues encountered in the preparation of those 2023.08 toolchains.
If you encounter any issue in the usage of those toolchains, or miss the support for a specific feature or architecture variant, let us know through the issue tracker. We hope those toolchains will continue to be useful to the community.
The DENT project is a project from the Linux Foundation which aims at utilizing the Linux Kernel, Switchdev, and other Linux based projects as the basis for building a new standardized network operating system without abstractions or overhead.
Recently, Bootlin collaborated with the DENT project to work on a specific topic: extending the Linux kernel NVMEM subsystem to be able to support the ONIE TLV storage format which is used on ONIE-compliant network equipment to store in an EEPROM various information about the device: serial number, model, MAC addresses, and more.
This work, lead by Bootlin engineer Miquèl Raynal has now landed in Linux 6.4 as the drivers/nvmem/layouts/onie-tlv.c driver, together with the underlying new NVMEM layout infrastructure, which Miquèl helped to upstream in collaboration with Michael Walle.
We have written and published a longer blog post on the DENT website to explain the motivation for this effort and the results.
As we reported in previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post, this one being the first of this series.
We are pleased to welcome two additional engineers to our team based in Toulouse, France: Romain Gantois and Louis Chauvet.
The 
Earlier this year, the 
