No less than 13 Bootlin engineers attended the Open Source Summit Europe 2024 in Vienna earlier this month, and two Bootlin engineers gave talks at this conference, keeping with Bootlin’s long standing tradition of sharing knowledge with our peers of the broader embedded Linux community.
In this blog post, we’re happy to share the slides and videos of the two talks we have during this event: Linux Power Management Features, Their Relationships and Interactions and Inspecting and Optimizing Memory Usage in Linux
On the week of September 16-20, Vienna is going the place to be for all open-source developers, with a large number of conferences taking place during this week. Being strong contributors to several key open-source projects, Bootlin will be strongly present at those conferences.
Open Source Summit Europe will take place on Sep 16-18, and its schedule has recently been published. This year, the Embedded Linux Conference part of this conference is unfortunately reduced to just 2 tracks of 2 days, but we nevertheless hope to see some interesting topics. Bootlin has contributed and will contribute to this event with:
Bootlin CEO Thomas Petazzoni being a member of the Embedded Linux Conference program committee, he helped reviewing and choosing the talks for the event
We look forward to meeting the community, presenting our projects and ideas and learning about the work done by others! See you all in Vienna in September!
As we reported in a previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog posts: first post, second post, third post, this one being the fourth and final post of the series.
Do the Time Warp – the Rocky Horror PTP Show: Verification of Network Time Synchronization in the Real World
Talk by Johannes Zink, chosen by Bootlin engineer Köry Maincent
As we are currently dealing with PTP at Bootlin and facing several weird behaviors, this talk resonated well with our current state of mind. Currently, most of our clock usage uses NTP but some specific usage may need PTP to have high-precision clock synchronization between devices.
In this talk, Johannes first describes briefly the principles of PTP and its implementation in the Linux kernel, where the PTP is either managed by the MAC (often), the PHY or by software, and Userspace, with the description of the Linuxptp project. Then he goes straight to the issues he faced. For non-PTP users, it might be a bit harsh to follow the tests and oscilloscope measurements described by Johannes. He describes several possible issues and clock behaviors you can face, which might help a new PTP user to not spend too much time on debugging some tricky PTP behavior. Also one of the important things he notices is to “Always check your assumptions!”, which he wants to spread as a religious mantra. Using his common pitfalls and best practices may be a good thing when putting a hand in the PTP mechanism.
And don’t forget “Always check your assumptions!”!
As a Yocto user, you may have already wondered, ‘Why aren’t there official tools for creating and managing BitBake-based projects in a reproducible manner?’ Perhaps you have already used tools like repo, Git submodules, kas, or even created your own scripts.
In this talk, Alexander Kanavin – one of the major contributors to the Yocto project – introduces the tools currently under development within OE-core/poky to address this situation.
WirePlumber 0.5 Propelling PipeWire for the Embedded
Talk by Ashok Sidipotu, chosen by Bootlin engineer Alexandre Belloni
Ashok started to present a quick introduction to what Pipewire is. A nice block diagram explains what it looks like in action. Then the discussion switches to the session manager and why it is important. WirePlumber is now the default session manager, replacing PipeWire media session. It manages the control path and dynamically creates PipeWire objects.
The main changes are:
config syntax is switching from Lua to SPA JSON, just like PipeWire. More info is available is this blog post
the event dispatcher has been created to handle PipeWire signals. This allows to prioritize signals and to avoid race conditions. This feature has a nice example and a fairly complete blog post
This talk is a nice overview of what is happening in the PipeWire ecosystem which is now quite mature. It is also great to see the improvements and that the embedded use case is not forgotten.
As we reported in a previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post: first post, second post, this one being the third of the series.
rtla timerlat: Debugging Real-time Linux Scheduling Latency
Talk by Daniel Bristot de Oliveira, chosen by Bootlin engineer Maxime Chevallier.
Talks related to real-time linux debugging are pretty common at ELCE, I gave one myself in 2017 and I’ve been attending most of them since then. Besides a headache, what I could get from attending all these talks is that this topic is complex, time consuming, and that there’s a lot of different methodologies one can use to find the cause of these elusive problems.
Users who aren’t very familiar with the inner workings of the Linux Kernel can ask for help on mailing-lists, and the reply usually asks for a trace. This is where things get complicated, the Linux kernel tracer is very powerful, but can drown users in a flood of trace events from which it is difficult to extract the relevant data.
Hopefully, Daniel’s talk is going to make this kind of talk less common, as the tool he wrote and presented, rtla, makes it easy to gather important information about the cause of undesired latencies. By using cleverly placed trace-points, in-kernel testing tools (timerlat and osnoise) and an automated trace analyzer, rtla can not only detect latencies as cyclictest would, it can also give you what caused the latency. If it’s a blocking problem, rtla tells you which process is blocking your task. If it’s an interference, rtla will tell you which task or interrupt caused the latency, and can even detect if the hardware itself is the culprit.
For developers, this tool is also a perfect way to gather user feedback and bug reports that are small, precise and easily reproducible.
I therefore strongly recommend checking out Daniel’s talk and his dedicated blog article.
Zbus – the Lightweight and Flexible Zephyr Message Bus
Talk by Robrigo Peixoto, chosen by Bootlin engineer Thomas Perrot
Zbus is a new message bus for Zephyr allowing threads to communicate to many others, easily. This bus allows to implement several bus topologies:
one-to-one
one-to-many
Many-to-many
In addition, it can be used on very constrained systems.
In this talk, Rodrigo explained in detail how Zbus works, through a few examples. A thread can read or publish in bus channels, and when a message is published into a channel:
The Listener’s callbacks are executed
A notification is put to the subscriber’s queues
Then the subscriber will be executed by priority order
The bus is managed by a dispatcher, named Virtual Distributed Event Dispatcher (VDED) that is robust to priority inversion.
We found Zbus to be a very interesting feature because before there was no easy way to implement one-to-many and many-to-many topologies, but also one-to-one communications without having to manage the problems of inverting priorities and to use FIFO, LIFO, pipe, etc.
Linux Power ! (from the Perspective of a PMIC Vendor)
Talk by Matti Vaittinen, chosen by Bootlin engineer Kamel Bouhara.
PMICs (Power Management Integrated Circuit) are a key component of low power embedded systems as they often handle complexity in controlling various power voltages required by SoCs. In his talk Matti Vaittinen started by depicting the various devices that can be embedded in a PMIC (Power Management Integrated Circuit): watchdog, RTC, GPIOs are examples of such extra functionalities. He reminded us the reason why such devices are best fitted in the Linux MFD subsystem to take advantage of existing code. However the main subsystem used to implement support for a PMIC is the regulator subsystem and the talk gives us a good understanding of how it works, the concept of provider/consumer, how to register multiple regulators for a PMIC and how to handle specific events. A focus is made on error detection and how over current errors are reported over three categories:
PROTECTION : hardware level errors reported when protection limit is reached
ERROR: Unrecoverable errors that don’t directly involve hardware shutdown.
WARNING: System is still recoverable but requires specific action to be taken
Some PMICs also provide IRQs to notify errors or events and the kernel provides a helper function to handle such notifications and map them to specific actions depending on their severity.
Overall, we found this talk interesting to understand bettert the features provided by PMICs, and how these features are supported by Linux.
As we reported in previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post: first post, this one being the second post.
As we reported in previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post, this one being the first of this series.
In the Embedded Linux ecosystem, the Embedded Linux Conference is the most important event, covering all topics related to the usage of Linux in embedded systems, and probably gathering the largest audience of embedded Linux developers and maintainers.
After several years where it was combined in the much larger Open Source Summit, mixed with conferences on largely unrelated topics, the Embedded Linux Conference is this year grouped only with other embedded-related conferences under an umbrella event called the Embedded Open Source Summit.
Like every year, Bootlin will have a strong participation to the event: no less than 14 engineers of our team will be at the conference, which is almost our entire team. At Bootlin, we strongly believe that participating to conferences is a key aspect of an engineer’s job, in order to stay up-to-date with the latest developments in our field, but also to make or strengthen connections with other members of the embedded Linux community.
Finally it is worth mentioning that Bootlin has already started contributing to the conference: as a member of the Embedded Linux Conference program committee, Bootlin CEO Thomas Petazzoni has already reviewed and participated to the selection of talks that made it to the schedule of this year’s conference.
Bootlin CEO Thomas Petazzoni is again a member of the program committee for this edition of ELCE, and has helped with other members of this committee in reviewing and selecting the numerous talk submissions that have been received.
Bootlin will obviously be present at this conference. With 13 engineers from Bootlin participating, almost the entire company will be in Dublin for this major event of the embedded Linux community. Also, 4 of the talks that we had submitted have been accepted:
Luca Ceresoli on Basics of I2C on Linux This talk is an introduction to using I²C on embedded Linux devices. I²C (or I2C) is a simple but flexible electronic bus to allow low-speed communication between the CPU and all sorts of chips: PMICs, ADC/DACs, GPIO expanders, video sensors, audio codecs, EEPROMS, RTCs and many more. It is so popular that knowing it is a must for any embedded system engineer. Luca will first give an introduction to what I2C is at the electrical level. He will then describe how I2C is implemented in the Linux kernel driver model and how that appears in sysfs, how to describe I2C devices using device tree and how to write a driver for an I2C device. Finally he will present the tools to communicate with the chips from userspace and share some debugging techniques, including inspection of the physical bus and software-level debugging.
Miquèl Raynal on Improving Wireless PAN Support Anybody eager to learn about IoT devices has at least once tried to play with Zigbee or 6lowpan sensors. These two protocols are built on top of a well common MAC/PHY specification: IEEE 802.15.4, also known as Wireless Personal Area Networks: WPAN, designed to be low-rate/low-range wireless networks. There is already substantial support for this protocol in the Linux kernel but when my journey started, several of the MAC-related operations well described in the specification were not implemented, making the subsystem mainly useful for very simple use cases: peer-to-peer transmissions. This is unfortunate as a significant part of the idea behind WPAN is to make these networks quite adaptive and resilient, which requires a minimal subset of the peer management procedure to be supported. Besides a number of preparation changes, the main idea behind the continuous flow of patches was to bring support for the scanning procedure which allows a PAN controller to detect all the compatible devices around it in different ways. Discovering these devices is the first step in order to associate them together and build up starred networks. This talk will be an opportunity to explain the new APIs allowing such discoveries and provide a state of the art of the support in the mainline kernel.
Michael Opdenacker on Implementing A/B System Updates with U-Boot A popular way to implement system updates is through the A/B scheme, in which you have two copies of the root filesystem, one which is active, and one that is meant to contain the next update. When a new update is successfully applied, you need to make the corresponding partition become the new active one. That’s when a number of practical questions arise, such as how to identify the active partition, how to detect when the new system fails to boot properly, and how to fall back to the previous version? It was hard to find documentation about how U-Boot could address such needs to implement a functional and failsafe A/B system update mechanism. This presentation proposes to address this need by sharing the practical solutions we found, using lesser known commands and capabilities in U-Boot. We will also explain how the Linux side can cooperate with the U-Boot side. Fortunately, you won’t need to erase half of your brain to get updated on this topic.
Paul Kocialkowski on Walking Through the Linux-Based Graphics Stack The graphics stack used with the Linux kernel is a notoriously complex beast. From userspace down to the kernel level, a number of components are involved and interact with eachother. It is also an area that is constantly evolving to meet new use cases, refresh legacy implementations and achieve better performance. This makes it difficult to have a clear idea of the big picture and what is actually happening when using graphics-related components. This presentation will detail a walk through the graphics stack, with actual examples of displaying a buffer and rendering using the GPU. Going from the application level through the system libraries, down to the kernel and ending with actual hardware configuration. State-of-the-art technologies such as Wayland and DRM will be highlighted with relevant excerpts from the source code of related free software projects that are widely used today.
We look forward to meeting again the embedded Linux community, its developers, users and maintainers, at Dublin during this conference!
The schedule for the next Embedded Linux Conference Europe has been recently published, and Bootlin will once again be strongly present at this (virtual) event by giving a number of presentations. The registration for ELC-E is open, and due to the virtual nature of the event, the registration cost is only $50, which makes is accessible to pretty much everybody.
From the Camera Sensor to the User, the Journey of a Video Frame. In this talk, Bootlin engineer Maxime Chevallier will share his experience working in the Video4Linux subsystem, implementing support for the Rockchip camera interface controller, and a PAL/NTSC decoder used as the input source. Talk on Monday October 26 at 18:30 GMT.
Yocto Project and OpenEmbedded: A Collection of Best Practices. In this talk, Bootlin engineer, and Yocto Project expert and trainer Alexandre Belloni will share his experience of using Yocto Project and OpenEmbedded through a collection of best practices. There are indeed numerous ways of using OpenEmbedded and the Yocto Project, but some ways and solutions are better than others! Talk on Tuesday October 27 at 13:00 GMT.
Building Embedded Debian and Ubuntu Systems with ELBE. In this talk, Köry Maincent will share his experience using the ELBE build system, which can be used to automate the process of creating embedded Linux systems based on Debian or Ubuntu. Köry has contributed to ELBE the support for building Ubuntu systems, and has used ELBE on two different projects. This is an interesting alternative to the traditional cross-compilation approach taken by Yocto Project, OpenEmbedded or Buildroot. Talk on Tuesday October 27 at 15:15 GMT.
Using Visual Studio Code for Embedded Development. In this talk, Michael Opdenacker will share his experiments of using VS Code for embedded Linux development, which extensions are available to help navigate in the code, build and debug code, from kernel-space and user-space. Talk on Tuesday October 27 at 17:15 GMT.
Precision Time Protocol (PTP) and Packet Timestamping in Linux. Antoine Ténart has implemented PTP support in both an Ethernet switch driver, and an Ethernet PHY driver, both in the upstream Linux kernel. He will share his experience about PTP, its support in Linux, and its offloading at the MAC and PHY level. Talk on Tuesday October 27 at 19:30 GMT.
Supporting Hardware-Accelerated Video Encoding with Mainline. After working on the HW-accelerated video decoding on Allwinner platforms as part of our crowdfunded effort, Paul Kocialkowski recently worked on HW-accelerated video encoding on Rockchip platforms. In this talk, he will share the issues encountered, and what needs to be resolved to create a useful kernel to userspace interface to properly support stateless video encoders. Talk on Wednesday October 28 at 16:15 GMT.
Understand ECC Support for NAND Flash Devices in Linux. Miquèl Raynal, the Linux kernel NAND subsystem maintainer, has recently worked on improving support for various strategies to handle ECC for NAND flash devices. He will share some background information on ECC, why they are needed, how and where ECC are typically handled, and how the Linux kernel deals with the different possibilities. Talk on Wednesday October 28 at 18:30 GMT.
In addition to contributing talks, Bootlin CTO Thomas Petazzoni is also a member of the ELC-E program committee: he reviewed, ranked all talk submitted for the conference and participated with the rest of the committee to the selection of the talks that are now scheduled for the event.
Even though we once again won’t have the chance to meet our fellow members of the embedded Linux community in person, we look forward to attending a set of great talks, and have interesting discussions during the Q&A and through the instant messaging platform that will be available around the conference.
With 8 engineers participating to the Embedded Linux Conference Europe, almost the entire Bootlin engineering team took part to the conference. As usual, we not only attended the event, but also contributed by giving a total of 5 talks and 2 tutorials, for which we’re happy to share below the videos and slides. Also, as part of this conference, Bootlin CTO Thomas Petazzoni received an award for his contribution to the conference.
Tutorial given by Michael Opdenacker, slides in PDF, slides source document. The video is not yet available, but should be published in the future.
Introduction to the Buildroot embedded Linux build system
Tutorial given by Thomas Petazzoni, slides in PDF, slides source code. The video is not yet available, but should be published in the future.
Award to Thomas Petazzoni
During the traditional closing game of the conference, we were really happy to have Bootlin’s CTO Thomas Petazzoni called on stage, to receive from the hands of Tim Bird, an award for his continuous 11 year participation to the conference, with 24 presentations given, one keynote and for the past two years, participation to the conference program committee. We are honored and proud by this recognition of Thomas contribution to the conference.
No less than 13 Bootlin engineers attended the Open Source Summit Europe 2024 in Vienna earlier this month, and two Bootlin engineers gave talks at this conference, keeping with Bootlin’s long standing tradition of sharing knowledge with our peers of the broader embedded Linux community.
In the Embedded Linux ecosystem, the 
The 
