The Embedded Open Source Summit 2024 took place on Apr 16-18 in Seattle, with many talks on a wide range of embedded Linux topics. 11 engineers from Bootlin participated to this conference and four of us gave talks, for which we are happy to publish the slides and videos in this blog post.
The Embedded Open Source Summit will take place on April 16-18 in Seattle (Washington, US), with a wide range of events, including the Embedded Linux Conference which is of particular interest to Bootlin considering our core expertise.
As usual, a significant fraction of the Bootlin team will attend the event, with no less than 11 engineers from our team going to Seattle: Alexandre Belloni, Bastien Curutchet, Grégory Clement, Jérémie Dautheribes, João Marcos Costa, Köry Maincent, Louis Chauvet, Luca Ceresoli, Maxime Chevallier, Thomas Petazzoni and an additional engineer who will join us at the beginning of April. At Bootlin, all engineers, regardless of their seniority level, are offered the possibility of attending up to 2 conferences per year, to expand their knowledge on embedded Linux topics, to create connections with the community and therefore provide better services to our customers.
In addition to participating, we will also be delivering no less than 4 talks at the conference:
In the Kernel Trenches: Mastering Ethernet Drivers on Linux – Maxime Chevallier, Bootlin An Ethernet Controller driver in linux is a complex piece of software, with many entrypoints and callbacks, even for a basic Ethernet Controller we can find in Embedded Systems. In this talk, we’ll take a tour of what’s inside a typical Ethernet driver. We’ll start with the base of what every network driver provides, even non-Ethernet ones, a net_device. We will discuss the various required steps to get a basic driver working, ranging from simple initialization, queue setup, DMA mappings, before diving into more advances topics such as the NAPI loop implementation, memory-management through the page-pool API and finally XDP, which is now getting implemented in more and more drivers. We’ll continue the tour through the ethernet-specific bits, such as the ethtool operations, and phylink support to interact with a potential Ethernet PHY, and discuss a little bit about the specifics of embedded Ethernet Controllers, where low-level configuration for a PCS or some Serdes lanes might be required. Finally, we’ll discuss about testing and debugging tools that can be useful to help you in your driver development, but also in debugging and optimizing existing drivers.
Introduction to DAPM, Linux Power Management for Embedded Audio Devices – Luca Ceresoli, Bootlin DAPM (Digital Audio Power Management) is the part of ASoC (Alsa for System-on-Chips) that dynamically turns on and off power to various parts of the audio hardware in embedded Linux systems. DAPM is very useful, but not well known and documented. After introducing the motivations for DAPM and how it models the hardware, Luca will present some user space tools to understand how DAPM works at runtime. He will then show what DAPM practically means for developers, with code examples of how to add DAPM support to device drivers. Finally he will mention some improvements to the DAPM tooling and documentation.
Unlocking the Potential of Suspend to RAM Using Linux in a Multi-Core, Multi-Firmware Automotive SoC – Grégory Clement, Bootlin Fast booting in modern, complex System-on-Chips (SoCs) demands innovative solutions. This talk delves into the intricacies of implementing Suspend to RAM (S2R) on a sophisticated SoC featuring diverse CPU cores—high-performance A72, real-time R5, and low-power M3—as well as multiple firmware components, including open-source ones like TF-A, U-Boot, and OP-TEE. We begin with a hardware overview of the TI DRA821 SoC from the Jacinto family, emphasizing the open-source firmware utilized in the boot process. The focus then shifts to presenting a practical approach for integrating S2R into this multifaceted platform. Notably, our implementation poses a unique challenge: the SoC is entirely powered off during S2R, adding an extra layer of complexity. Join us as we share insights into the hurdles encountered and resolved during the implementation, along with a discussion on the outstanding challenges for which we’ve devised viable solutions. This session offers a comprehensive exploration into the software, including bootloaders and the Linux kernel, as well as the hardware aspects, showcasing a real-world approach to bringing Suspend to RAM to life on a cutting-edge automotive SoC.
Adding Support for Power Over Ethernet (PoE or IEEE Clause 33) to Linux Network Stack – Köry Maincent, Bootlin Power over Ethernet (PoE) is a technology that combines electrical power and data transmission over a single Ethernet cable. It eliminates the need for separate power sources, simplifying installations for devices like IP cameras, and VoIP phones. In this presentation, we’ll initially delve into Power over Ethernet (PoE), It debuted in IEEE Clause 33 without explicit reference to the PoE nomenclature. We will look at what currently exists in the Linux kernel and user space to support PoE. We’ll continue our discussion with some details of the Linux implementation currently in development and the PSE core changes brought by this new support. The PSE framework core and bindings happen to need modification as it was not prepared for the PoE specificities. In parallel to these extensions of the PSE framework, we developed Linux kernel drivers for two distinct PoE controllers: the Microchip PD692x0 and the Texas Instruments TPS23881. Finally, we will look into the mainline status, the things that still need to be merged and the future features that need development. This PoE Linux support is sponsored and funded by DENT Project.
The Embedded Open Source Summit isn’t just about the presentations—it’s about forging connections, sharing ideas, and exploring opportunities. Join us for insightful conversations with the Bootlin team, where you can discuss your projects, ambitions, and maybe even discover new career pathways.
And mark your calendars because following the summit, our CEO Thomas Petazzoni will be in the Bay Area until April 24, ready to delve deeper into your embedded Linux needs.
So, whether you’re a seasoned veteran or a curious newcomer, don’t miss out on the Embedded Open Source Summit 2024. We’ll see you there!
At Bootlin, we focus on Embedded Linux development and support, and these embedded devices often have a network interface, be-it an Ethernet port, a Wireless chip or some other kind of communication channel that falls under the Linux Networking Stack’s framework.
So it’s always interesting to see what the rest of the community is working on, and meet in real life people we interact with on the netdev mailing list.
That’s why this year, Alexis Lothoré and Maxime Chevallier flew to Vancouver to participate to the Netdev Conference, a 5 days event organised by the Netdev Society, a small non-profit run by volunteers dedicated to holding this event.
Most talks at Netdev are not directly covering topics we’re actively working on, but it’s always refreshing to see these new exciting technologies that could trickle their way down to the embedded world a few years from now. It is also always pretty interesting to stay up to date about challenges encountered by other parts of the networking industry, at scales way different than the ones we are used to.
We learned for example what CXL is about, what it brings and the effort that are made to design new networking hardware around this technology to change the way we think about datacenter networking.
When we attended Netdev 0x13 in 2019, QUIC was one of the hot topics. This year, Homa was under the spotlight with talks on what it is, and how this new protocol could address some of TCP’s problems.
Like all previous editions, we learned all the progress that were made with TC and its future, new ways of bypassing the kernel stack, BPF integration in the kernel, along with XDP which continued to be more and more powerful.
Another hot topic in the kernel is the introduction of the Rust language, and the network subsystem is a pretty relevant target for the new features brought by the language. As a consequence, Rust subsystem maintainers Miguel Ojeda and Wedson Almeida Filho gave an overview of Rust benefits compared to traditional C code, and then showed a step-by-step implementation of a kernel-side TCP server module. While this example is not perfectly representative of classic network-related drivers we usually write, it was a nice showcase of current state of kernel APIs abstractions in Rust.
We also discovered the new use-case that is now driving most of the datacenter networking efforts, which is without surprise AI and Machine Learning. Turns out, if you want your ChatGPT to answer up-to-date replies without having to wait for too long, you need a powerful and well-organized datacenter for the training part, and networking engineering takes a big part in it to keep all those GPUs fed at a relevant pace.
This lead to the devmem TCP effort, which started to feel a bit familiar for us as it uses dma-buf, which we also sometimes use on multimedia pipelines. The ML and AI topic was introduced to us by the wonderful Keynote session given by Manya Ghobadi, who got all the audience captivated by how AI and ML works, what AI workloads requires in terms of network traffic scheduling, datacenter topology and computing hardware that uses optical computing.
On the final day, we even had a visit from Jakub Kicinski (one of the co-maintainers of Linux networking tree), presenting what he had been working on, and gave us an update on the netdev development statistics (and basically, his main point is that we do need to review more patches).
For the first time, there was a talk from Bootlin at netdev, as Maxime presented one of the topics he’s been working on lately : Improving multi-PHY and multi-port interfaces support. Although it was one of the only talks focusing on the low-levels aspects of the Ethernet stack, it triggered some discussions and interest from the community, which will help further improving the ongoing work.
The slides and videos of the event will be published at some point in the future, we will for sure mention this to our readers when it becomes available.
We’ll conclude this short feedback by thanking once again the Netdev Board members, organizers, speakers and the audience for this great event.
From June 28 to June 30, Bootlin participated to the Embedded Linux Conference Europe, which was organized as part of the new and larger Embedded Open Source Summit.
In addition, the day before the conference, on June 27, our team had a great team building event, spending the day visiting Prague, having lunch in a traditional restaurant, enjoying a boat tour on the Vltava river, and an evening with a traditional dinner and folklore music. As our team is distributed, conferences are a great opportunity to meet each other and Prague was for several members of our team their first in-person meeting.
With 14 Bootlin engineers at the conference, almost our entire engineering team participated. Indeed, we have a policy at Bootlin to offer to all our engineers, regardless of their seniority level, the chance of attending 2 technical conferences each year.
In the Embedded Linux ecosystem, the Embedded Linux Conference is the most important event, covering all topics related to the usage of Linux in embedded systems, and probably gathering the largest audience of embedded Linux developers and maintainers.
After several years where it was combined in the much larger Open Source Summit, mixed with conferences on largely unrelated topics, the Embedded Linux Conference is this year grouped only with other embedded-related conferences under an umbrella event called the Embedded Open Source Summit.
Like every year, Bootlin will have a strong participation to the event: no less than 14 engineers of our team will be at the conference, which is almost our entire team. At Bootlin, we strongly believe that participating to conferences is a key aspect of an engineer’s job, in order to stay up-to-date with the latest developments in our field, but also to make or strengthen connections with other members of the embedded Linux community.
Finally it is worth mentioning that Bootlin has already started contributing to the conference: as a member of the Embedded Linux Conference program committee, Bootlin CEO Thomas Petazzoni has already reviewed and participated to the selection of talks that made it to the schedule of this year’s conference.
Almost the entire engineering team of Bootlin attended the Embedded Linux Conference Europe 2022 in Dublin mid-september, an important event for Bootlin as it helps everyone in the team stay up to date with the latest developments in the Embedded Linux ecosystem, and connect with members of the community.
After such conferences, we have a tradition at Bootlin: share with our readers a selection of talks that we found interesting. Several members of our engineering team were asked to select one of their favorite talks, and highlight it with a short summary.
The schedule for the next edition of Live Embedded Event has been published! This 100% online and free conference will take place on June 3rd, 2021. Thanks to the proposals received, the event will feature 4 tracks during the entire day, covering a wide range of topics: hardware for embedded systems, embedded Linux, RTOS, IoT, FPGA, RISC-V, and more.
Bootlin is once again part of the organization team for this event, and in addition 5 talks proposed by Bootlin have been selected into the schedule. See below the details of our talks.
Understanding U-Boot Falcon Mode and adding support for new boards, Michael Opdenacker
The Falcon Mode is a U-Boot feature that allows to directly load the operating system kernel from the first stage of U-Boot (a.k.a. “SPL”), skipping the second stage of U-Boot. Doing this can save up to 1 second in the boot process, and this way, you can keep a full featured U-Boot that you can still fall back to for maintenance or development needs. However, using Falcon Mode is not always easy, as it requires extra code that most boards supported by U-Boot don’t have yet. At Bootlin, we had to add such support to U-Boot for several boards. This presentation will explain how Falcon Mode booting actually works in U-Boot and the implementation and usage choices made by U-Boot developers. It will show you how to add such Falcon Mode support to U-Boot for your own board.
In embedded systems, deploying firmware updates in the field has now become an obvious requirement, to ensure that security vulnerabilities are addressed, that bugs are fixed, and new functionalities can be delivered to the users. Among a range of different open-source solutions, RAUC provides an interesting firmware update mechanism for embedded system. In this talk, we will introduce the main features of RAUC, its integration in build systems such as Buildroot or the Yocto Project, as well as its integration with the U-Boot and Barebox bootloaders. Finally we will explore some common update scenarios that are fully supported by RAUC features.
Talk given by Kamel Bouhara, at 3:30 PM CEST on June 3rd, 2021.
Security vulnerability tracking tools in Buildroot, Thomas Petazzoni
Buildroot is a popular and easy to use embedded Linux build system. With the increasing concern around security vulnerabilities affecting embedded systems, and the need to keep them updated, Buildroot has been extended with new tooling for security vulnerability tracking. This tooling allows to monitor the CVEs that affect the packages present in Buildroot. In this talk, we will introduce the principle of CVEs and CPEs, present the tools now available in Buildroot to help keep track of the security vulnerabilities, show how they can be used for a project and identify the current limitations of this tooling.
Talk given by Thomas Petazzoni, at 1:30 PM CEST on June 3rd, 2021.
Secure boot in embedded Linux systems, Thomas Perrot
Secure boot is a integrity mechanism, based on signature verification, that allows to detect software corruption or malicious code, during the boot process. Implementing secure boot is not always obvious, as it requires multiple stages of verification, at the bootloader, Linux kernel and root filesystem level, as well as integration into the build system, CI infrastructure, firmware upgrade mechanism, and more. Based on a recent experience to bring secure boot on an NXP i.MX8 platform, Thomas will present how to implement the chain of trust from the SoC ROM code to the root filesystem, as well as other considerations related to the implementation of secure boot. While the presentation will use the i.MX8 as an example, most of the discussion will apply to other platforms as well.
Talk given by Thomas Perrot, at 3:30 PM CEST on June 3rd, 2021.
Device Tree overlays and U-boot extension board management, Köry Maincent
In this talk, we will start by introducing the mechanism of Device Tree Overlays, which are a way of extending the Device Tree itself to describe additional hardware. We will show how Device Tree Overlays are written, compiled, and applied to a base Device Tree, and what is the status of Device Tree Overlays support in U-Boot and Linux. We will take the example of the BeagleBoard.org project, showing how Device Tree overlays are used to make CAPE extension boards compatible with different boards. Finally, we will describe our proposal, already submitted to the community, to add an extension board management facility to U-Boot, which automatically detects, loads and applies the appropriate Device Tree Overlays depending on the extension boards that are detected.
Talk given by Köry Maincent, at 1:30 PM CEST on June 3rd, 2021.
The videos from Bootlin’s presentations earlier this month at FOSDEM 2021 are now publicly available. Once again, FOSDEM was a busy event, even if it was online for once. As in most technical conferences, Bootlin engineers volunteered to share their experience and research by giving two talks.
Maxime Chevallier – Network Performance in the Linux Kernel, Getting the most out of the Hardware
Abstract: The networking stack is one of the most complex and optimized subsystems in the Linux kernel, and for a good reason. Between the wild range of applications, the complexity and variety of the networking hardware, getting good performance while keeping the stack easily usable from userspace has been a long-standing challenge.
Nowadays, complex Network Interface Controllers (NICs) can be found even on small embedded systems, bringing powerful features that were previously found only in the server world closest to day to day users.
This is a good opportunity to dive into the Linux Networking stack, to discover what is used to make networking as fast as possible, both by using all the features of the hardware and by implementing some clever software tricks.
In this talk, we cover these various techniques, ranging from simple batch processing with NAPI, queue management with RSS, RPS, XPS and so on, flow steering and filtering with ethool and TC, to finish with the newest big change that is XDP.
We dive into these various techniques and see how to configure them to squeeze the most out of your hardware, and discover that what was previously in the realm of datacenters and huge computers can now also be applied to embedded linux development.
Michael Opdenacker – Embedded Linux from Scratch in 45 minutes, on RISC-V
Abstract: Discover how to build your own embedded Linux system completely from scratch. In this presentation and tutorial, we show how to build a custom toolchain (Buildroot), bootloader (opensbi / U-Boot) and kernel (Linux), that one can run on a system with the new RISV-V open Instruction Set Architecture emulated by QEMU. We also show how one can build a minimal root filesystem by oneself thanks to the BusyBox project. The presentation ends by showing how to control the system remotely through a tiny webserver. The approach is to provide only the files that are strictly necessary. That’s all the interest of embedded Linux: one can really control and understand everything that runs on the system, and see how simple the system can be. That’s much easier than trying to understand how a GNU/Linux system works from a distribution as complex as Debian!
The presentation also shares details about what’s specific to the RISC-V architecture, in particular about the various stages of the boot process. This presentation shares all the hardware (!), source code build instructions and demo binaries needed to reproduce everything at home, and add specific improvements. Most of the details are also useful to people using other hardware architectures (in particular arm and arm64).
It’s probably the first time a tutorial manages to show so many aspects of embedded Linux in less than an hour. See by yourself! At least, that’s for sure the first one demonstrating how to boot Linux from U-Boot in a RISC-V system emulated by QEMU.
Like all conferences in these times, FOSDEM will take place as an online, virtual event. For all the FOSDEM regular attendees, it will certainly be a very different experience, and for sure, we will all miss the chocolate, waffles, beer, mussels as well as the rainy, muddy, snowy, foggy and cold weather that characterize Brussels in early February. But nevertheless, knowledge sharing and discussions must go on, and FOSDEM will take place! As usual, FOSDEM takes place the first week-end of February, on February 6-7, and the event is completely free, with no registration required.
This time around, Bootlin is once again contributing to FOSDEM:
Thomas Petazzoni is a member of the program committee for the Embedded, Mobile and Automotive Devroom. As such, he has worked with the other members of this program committee to select the talks, define the schedule, and will help organize the talks during FOSDEM itself.
The Embedded Linux Conference Europe took place online last week. While we definitely missed the experience of an in-person event, we strongly participated to this conference with no less than 7 talks on various topics showing Bootlin expertise in different fields: Linux kernel development in networking, multimedia and storage, but also build systems and tooling. We’re happy to be publishing now the slides and videos of our talks.
From the camera sensor to the user: the journey of a video frame, Maxime Chevallier