From June 28 to June 30, Bootlin participated to the Embedded Linux Conference Europe, which was organized as part of the new and larger Embedded Open Source Summit.
In addition, the day before the conference, on June 27, our team had a great team building event, spending the day visiting Prague, having lunch in a traditional restaurant, enjoying a boat tour on the Vltava river, and an evening with a traditional dinner and folklore music. As our team is distributed, conferences are a great opportunity to meet each other and Prague was for several members of our team their first in-person meeting.
With 14 Bootlin engineers at the conference, almost our entire engineering team participated. Indeed, we have a policy at Bootlin to offer to all our engineers, regardless of their seniority level, the chance of attending 2 technical conferences each year.
In the Embedded Linux ecosystem, the Embedded Linux Conference is the most important event, covering all topics related to the usage of Linux in embedded systems, and probably gathering the largest audience of embedded Linux developers and maintainers.
After several years where it was combined in the much larger Open Source Summit, mixed with conferences on largely unrelated topics, the Embedded Linux Conference is this year grouped only with other embedded-related conferences under an umbrella event called the Embedded Open Source Summit.
Like every year, Bootlin will have a strong participation to the event: no less than 14 engineers of our team will be at the conference, which is almost our entire team. At Bootlin, we strongly believe that participating to conferences is a key aspect of an engineer’s job, in order to stay up-to-date with the latest developments in our field, but also to make or strengthen connections with other members of the embedded Linux community.
Finally it is worth mentioning that Bootlin has already started contributing to the conference: as a member of the Embedded Linux Conference program committee, Bootlin CEO Thomas Petazzoni has already reviewed and participated to the selection of talks that made it to the schedule of this year’s conference.
Almost the entire engineering team of Bootlin attended the Embedded Linux Conference Europe 2022 in Dublin mid-september, an important event for Bootlin as it helps everyone in the team stay up to date with the latest developments in the Embedded Linux ecosystem, and connect with members of the community.
After such conferences, we have a tradition at Bootlin: share with our readers a selection of talks that we found interesting. Several members of our engineering team were asked to select one of their favorite talks, and highlight it with a short summary.
The schedule for the next edition of Live Embedded Event has been published! This 100% online and free conference will take place on June 3rd, 2021. Thanks to the proposals received, the event will feature 4 tracks during the entire day, covering a wide range of topics: hardware for embedded systems, embedded Linux, RTOS, IoT, FPGA, RISC-V, and more.
Live Embedded Event #2 agenda
Bootlin is once again part of the organization team for this event, and in addition 5 talks proposed by Bootlin have been selected into the schedule. See below the details of our talks.
Understanding U-Boot Falcon Mode and adding support for new boards, Michael Opdenacker
The Falcon Mode is a U-Boot feature that allows to directly load the operating system kernel from the first stage of U-Boot (a.k.a. “SPL”), skipping the second stage of U-Boot. Doing this can save up to 1 second in the boot process, and this way, you can keep a full featured U-Boot that you can still fall back to for maintenance or development needs. However, using Falcon Mode is not always easy, as it requires extra code that most boards supported by U-Boot don’t have yet. At Bootlin, we had to add such support to U-Boot for several boards. This presentation will explain how Falcon Mode booting actually works in U-Boot and the implementation and usage choices made by U-Boot developers. It will show you how to add such Falcon Mode support to U-Boot for your own board.
In embedded systems, deploying firmware updates in the field has now become an obvious requirement, to ensure that security vulnerabilities are addressed, that bugs are fixed, and new functionalities can be delivered to the users. Among a range of different open-source solutions, RAUC provides an interesting firmware update mechanism for embedded system. In this talk, we will introduce the main features of RAUC, its integration in build systems such as Buildroot or the Yocto Project, as well as its integration with the U-Boot and Barebox bootloaders. Finally we will explore some common update scenarios that are fully supported by RAUC features.
Talk given by Kamel Bouhara, at 3:30 PM CEST on June 3rd, 2021.
Security vulnerability tracking tools in Buildroot, Thomas Petazzoni
Buildroot is a popular and easy to use embedded Linux build system. With the increasing concern around security vulnerabilities affecting embedded systems, and the need to keep them updated, Buildroot has been extended with new tooling for security vulnerability tracking. This tooling allows to monitor the CVEs that affect the packages present in Buildroot. In this talk, we will introduce the principle of CVEs and CPEs, present the tools now available in Buildroot to help keep track of the security vulnerabilities, show how they can be used for a project and identify the current limitations of this tooling.
Talk given by Thomas Petazzoni, at 1:30 PM CEST on June 3rd, 2021.
Secure boot in embedded Linux systems, Thomas Perrot
Secure boot is a integrity mechanism, based on signature verification, that allows to detect software corruption or malicious code, during the boot process. Implementing secure boot is not always obvious, as it requires multiple stages of verification, at the bootloader, Linux kernel and root filesystem level, as well as integration into the build system, CI infrastructure, firmware upgrade mechanism, and more. Based on a recent experience to bring secure boot on an NXP i.MX8 platform, Thomas will present how to implement the chain of trust from the SoC ROM code to the root filesystem, as well as other considerations related to the implementation of secure boot. While the presentation will use the i.MX8 as an example, most of the discussion will apply to other platforms as well.
Talk given by Thomas Perrot, at 3:30 PM CEST on June 3rd, 2021.
Device Tree overlays and U-boot extension board management, Köry Maincent
In this talk, we will start by introducing the mechanism of Device Tree Overlays, which are a way of extending the Device Tree itself to describe additional hardware. We will show how Device Tree Overlays are written, compiled, and applied to a base Device Tree, and what is the status of Device Tree Overlays support in U-Boot and Linux. We will take the example of the BeagleBoard.org project, showing how Device Tree overlays are used to make CAPE extension boards compatible with different boards. Finally, we will describe our proposal, already submitted to the community, to add an extension board management facility to U-Boot, which automatically detects, loads and applies the appropriate Device Tree Overlays depending on the extension boards that are detected.
Talk given by Köry Maincent, at 1:30 PM CEST on June 3rd, 2021.
The videos from Bootlin’s presentations earlier this month at FOSDEM 2021 are now publicly available. Once again, FOSDEM was a busy event, even if it was online for once. As in most technical conferences, Bootlin engineers volunteered to share their experience and research by giving two talks.
Maxime Chevallier – Network Performance in the Linux Kernel, Getting the most out of the Hardware
Abstract: The networking stack is one of the most complex and optimized subsystems in the Linux kernel, and for a good reason. Between the wild range of applications, the complexity and variety of the networking hardware, getting good performance while keeping the stack easily usable from userspace has been a long-standing challenge.
Nowadays, complex Network Interface Controllers (NICs) can be found even on small embedded systems, bringing powerful features that were previously found only in the server world closest to day to day users.
This is a good opportunity to dive into the Linux Networking stack, to discover what is used to make networking as fast as possible, both by using all the features of the hardware and by implementing some clever software tricks.
In this talk, we cover these various techniques, ranging from simple batch processing with NAPI, queue management with RSS, RPS, XPS and so on, flow steering and filtering with ethool and TC, to finish with the newest big change that is XDP.
We dive into these various techniques and see how to configure them to squeeze the most out of your hardware, and discover that what was previously in the realm of datacenters and huge computers can now also be applied to embedded linux development.
Michael Opdenacker – Embedded Linux from Scratch in 45 minutes, on RISC-V
Abstract: Discover how to build your own embedded Linux system completely from scratch. In this presentation and tutorial, we show how to build a custom toolchain (Buildroot), bootloader (opensbi / U-Boot) and kernel (Linux), that one can run on a system with the new RISV-V open Instruction Set Architecture emulated by QEMU. We also show how one can build a minimal root filesystem by oneself thanks to the BusyBox project. The presentation ends by showing how to control the system remotely through a tiny webserver. The approach is to provide only the files that are strictly necessary. That’s all the interest of embedded Linux: one can really control and understand everything that runs on the system, and see how simple the system can be. That’s much easier than trying to understand how a GNU/Linux system works from a distribution as complex as Debian!
The presentation also shares details about what’s specific to the RISC-V architecture, in particular about the various stages of the boot process. This presentation shares all the hardware (!), source code build instructions and demo binaries needed to reproduce everything at home, and add specific improvements. Most of the details are also useful to people using other hardware architectures (in particular arm and arm64).
It’s probably the first time a tutorial manages to show so many aspects of embedded Linux in less than an hour. See by yourself! At least, that’s for sure the first one demonstrating how to boot Linux from U-Boot in a RISC-V system emulated by QEMU.
Like all conferences in these times, FOSDEM will take place as an online, virtual event. For all the FOSDEM regular attendees, it will certainly be a very different experience, and for sure, we will all miss the chocolate, waffles, beer, mussels as well as the rainy, muddy, snowy, foggy and cold weather that characterize Brussels in early February. But nevertheless, knowledge sharing and discussions must go on, and FOSDEM will take place! As usual, FOSDEM takes place the first week-end of February, on February 6-7, and the event is completely free, with no registration required.
This time around, Bootlin is once again contributing to FOSDEM:
Thomas Petazzoni is a member of the program committee for the Embedded, Mobile and Automotive Devroom. As such, he has worked with the other members of this program committee to select the talks, define the schedule, and will help organize the talks during FOSDEM itself.
The Embedded Linux Conference Europe took place online last week. While we definitely missed the experience of an in-person event, we strongly participated to this conference with no less than 7 talks on various topics showing Bootlin expertise in different fields: Linux kernel development in networking, multimedia and storage, but also build systems and tooling. We’re happy to be publishing now the slides and videos of our talks.
From the camera sensor to the user: the journey of a video frame, Maxime Chevallier
In these times of COVID19, pretty much all of the existing conferences have moved to an online format. For example, the Embedded Linux Conference Europe is going to take place next week, online, and Bootlin will significantly contribute to the event with no less than 7 talks on a wide range of topics.
But this trend for online conferences has also spurred the creation of new events. And specifically, we’re happy to announce the creation of a new conference oriented towards our favorite topic of embedded systems: Live Embedded Event. It will take place online on December 3 and will have a broader range of topics covered than ELC typically has, as Live Embedded Event is open to non-Linux embedded topics, hardware platform and interfaces discussions, and more.
The event is mainly organized by four French companies specialized in embedded systems, and we expect the contents to be a mix of talks in English and in French. Speaking of contents, the Call For Papers is open until November 10, so you can have the chance to share your knowledge about embedded topics, show your expertise, and meet new users and customers. And of course, you can also simply register as an attendee: the event is free !
The Netdev 0x13 conference took place last week in Prague, Czech Republic. As we work on a variety of networking topics as part of our Linux kernel contributions, Bootlin engineers Maxime Chevallier and Antoine Ténart went to meet with the Linux networking community and to see a lot of interesting sessions. It’s the third time we enjoy attending the Netdev conference (after Netdev 2.1 and Netdev 2.2) and as always, it was a blast!
The 3-day conference started with a first day of workshops and tutorials. We enjoyed learning how to be the cool kids thanks to the XDP hands-on tutorial where Jesper Brouer and Toke Høiland-Jørgensen cooked us a number of lessons to progressively get to learn how to write and load XDP programs. This was the first trial-run of the tutorial which is meant to be extended and used as a material to go through the XDP basics. The instructions are all available on Github.
We then had the chance to attend the TC workshop where face to face discussions and presentations of the traffic control hot topics being worked on happened. The session caught our attention as the topic is related to current subjects being worked on at Bootlin.
Being used to work on embedded systems, seeing the problems the Network developers face can sometimes come as a surprise. During the TC workshop, Vlad Buslov presented his recent work on removing TC flower’s the dependency to the global rtnl lock, which is an issue when you have a million classification rules to update quickly.
We also went to the hardware offload workshop. The future of the network offload APIs and support in the Linux kernel was discussed, with various topics ranging from ASIC support to switchev advanced use-cases or offloading XDP. This was very interesting to us as we do work on various networking engines providing many offloading facilities to the kernel.
The next two days were a collection of talks presenting the recent advances in the networking subsystem of the Linux kernel, as well as current issues and real-world examples of recent functionalities being leveraged.
As always XDP was brought-up with a presentation of XDP offloading using virtio-net, recent advances in combining XDP and hardware offloading techniques and a feedback from Cloudflare using XDP in their DDOS mitigation in-house solution.
But we also got to see other topics, such as SO_TIMESTAMPING being used for performance analytics. In this talk Soheil Hassas Yeganeh presented how the kernel timestamping facilities can be used to track individual packets withing the networking stack for performance analysis and debugging. This was nice to see as we worked on enabling hardware timestamping in networking engines and PHYs for our clients.
Another hot topic this year was the QUIC protocol, which was presented in details in the very good QUIC tutorial by Jana Iyengar. Since this protocol is fairly new, it was brought-up in several sessions from a lot of interesting angles.
Although QUIC was not the main subject of Alissa Cooper’s keynote on Open Source, the IETF, and You, she explained how QUIC was an example of a protocol that is designed alongside its implementations, having a tight feedback loop between the protocol specifications and its usage in real-life. Alissa shared Jana’s point on how middle-boxes are a problem when designing and deploying new protocols, and explained that an approach to overcome this “ossification” is to encrypt the protocol header themselves and document the invariant parts of the non-encrypted parts.
A consequence of having a flexible protocol is that it is not meant to be implemented in the kernel. However, Maciej Machnikowski and Joshua Hay explained that it is still possible to offload some of the processing to hardware, which sparked interesting discussions with the audience on how to do so.
Conclusion
The Netdev 0x13 conference was well organized and very pleasant to attend. The content was deeply technical and allowed us to stay up-to-date with the latest developments. We also had interesting discussions and came back with lots of ideas to explore.
Thanks for organizing Netdev, we had an amazing time!
It’s now a tradition: the Buildroot project organizes one of its Buildroot Developers Meeting right after the FOSDEM conference. 2019 was no exception, and the meeting took place from February 4 to February 6, a three days duration, instead of the traditional two days duration from the previous years. Once again, the meeting was sponsored by Google, who provided the meeting location and lunch for all participants. Bootlin participated to the event, by allowing its engineer Thomas Petazzoni to join the meeting.
The meeting was a mix of discussions on various topics and actual hacking, with a focus on reducing the backlog of pending patches. The report synthetizes the most important discussion items:
Some discussions around the download infrastructure took place, related to the re-introduction of the make source-check feature and the issue of tarball reproducibility with version control system download backends
Discussion about introducing Config.in options for all host packages, an idea that we decided to not pursue for the moment.
Discussion about the instrumentation hooks that are used to collect the list of files installed by packages, and how we can achieve this goal in a way that is both efficient and reliable
Discussion on which Qt5 versions to support
Discussion on participating to the Google Summer of Code. We wrote a few topic ideas and applied as an organization for GSoC 2019.
Discussion on how to integrate support for systemd sysusers mechanism
Reading the work on the pending patches, we managed to reduce the backlog from about 300 patches to around 170 patches, which is a very significant achievement.
More specifically, Thomas Petazzoni took advantage of this meeting to:
Finalize his work on the pkg-stats script, to include details about the latest available upstream version of each Buildroot package. To do so, it relies on information provided by the release-monitoring.org website. The information is not yet accurate for all packages, but the accuracy can be improved by contributing to release-monitoring.org. The updated package statistics page now provides those details, which will help ensure Buildroot packages are kept up-to-date.
Review in detail the patch series from Adam Duskett introducing support for GObject Introspection. It is far from a trivial package due to the need to run during the build some small binaries using Qemu. While the series is not merged yet, we have a much better understanding of it, which will help complete the review process.
Do a final review and apply the lengthy patch series reworking the fftw package.
Participate, as a Buildroot co-maintainer, to the pending patches backlog cleanup, by reviewing and/or merging a significant number of patches.
It was once again a very nice and productive meeting. The next meeting will take place as usual around the Embedded Linux Conference Europe, in October, in Lyon (France).
In the Embedded Linux ecosystem, the 
Almost the entire engineering team of Bootlin attended the 
It’s now a tradition: the 