Almost the entire engineering team of Bootlin attended the Embedded Linux Conference Europe 2022 in Dublin mid-september, an important event for Bootlin as it helps everyone in the team stay up to date with the latest developments in the Embedded Linux ecosystem, and connect with members of the community.
All the slides and videos are available at https://elinux.org/ELC_Europe_2022_Presentations, which is one of the great things about the Embedded Linux Conference.
After such conferences, we have a tradition at Bootlin: share with our readers a selection of talks that we found interesting. Several members of our engineering team were asked to select one of their favorite talks, and highlight it with a short summary.
Talk “What Does a CPU Do Before Going to Work (in an Embedded System)?” by Josef Holzmayr
Talk selected by Grégory Clement
This talk was about the boot process, something we know pretty well but it was interesting to see how it was explained. The reason why we selected this talk is because we found it was at the same time very pertinent and accurate.
During his talk Josef compared the boot process of a CPU to someone waking-up to go working. This comparison was very effective and pleasant to follow. He also managed to present all kinds of situation from basic MCUs to very complex modern SoCs.
For anyone who is not very familiar or comfortable with the boot process, this talk will provide a good and easy to remember representation.
Talk “How to get your Devicetree schema bindings accepted in less than 10 iterations” by Krzysztof Kozlowski
Talk selected by Hervé Codina
As we already had some issue with upstreaming device-tree bindings, we were interested in Krzysztof’s talk related to this topic.
Krzysztof is the co-maintainer of the Device Tree bindings. He started his presentation with a brief summary of the Device Tree purpose (describe the hardware) and the bindings goal (describe the dts Device Tree sources). He presented the new DT schema format used for Device Tree bindings. This new format allows to validate the bindings themselves and the DTS file against the bindings.
Then, he described some simple examples of Device Tree sources and DT schema and he showed the minimal items that are expected in the DT schema. He mentioned the tools make dt_binding_check and make dtbs_check used to validate the bindings and the Device Tree source files themselves.
He also presented what we should and shouldn’t do when writing a new bindings. The “does and dont’s” slides offer great recommendations and examples about some properties related to prefix usage, names, units, types, arrays with or without minItems nor maxItems and some other things related to basic properties. He described also the purpose and the usage of additionalProperties / unevaluatedProperties.
The part of the talk titled “Reusable patterns” showing classical DT schema patterns used (and reusable) in several bindings was a great part. Indeed, the patterns presented here can be used to solve some use-cases for new DT schema such as conditional parts based on the compatible string.
These talk slides were some kind of “must-read” slides in order to have very good references about DT schema. The talk contains also a good and instructive ‘Questions and Answers’ session.
Talk “What’s Missing in Embedded Build Systems” by Arnout Vandecappelle
Talk selected by Köry Maincent
At Bootlin, we regularly do some similar work on each configuration of our BSP projects like Read-Only Overlay configuration or System Update, and this talk’s title and abstract consequently resonated well with what we’re doing.
Arnout is a Buildroot maintainer. He works on a lot of different BSPs therefore he has also seen these recurrent tasks. His talk is presenting these different tasks that are often needed and for which we often reinvent the wheel every time.
He looked at how these recurrent configurations are not handled by build systems and what could be done on Buildroot or Yocto about it. He agreed that each project can use its preferred solution but proposing a default solution could be a great idea to not lose time rewriting them every time. Indeed by taking the system update as an example we can choose between several update clients (SWUpdate, RAUC, Mender, etc.) and having this choice is good, but we need to rewrite the configuration every time even if they are similar. Having a default system update already configured, without any dedicated modification, will be time-saving but we need to let the community decide the right choice between them and the default update implementation.
Here is the list of the topics where we are reinventing the wheel each time on the build systems:
- Field Update
- Using persistent partition alongside Read-only Rootfs to save configuration and data
- Factory reset
- Manufacturing flash
- Provisioning per device data (MAC, IDs, serial, certificates …), which need to be saved persistently but not removed during a Factory reset
- Verified boot. Need standard choices for each boot step.
We can see that we have really useful and reliable build systems but it seems there are still a few items where we reinvent the wheel every time. It could be great to raise debate in the community about it and find default solutions (with maybe a few alternatives alongside) and propose an easy and time-saving way to tackle these items.
Talk “RAUC: (R)evolution of an Update Framework” by Enrico Jörns, Pengutronix
Talk selected by Luca Ceresoli
Enrico gave a great overview of the current status of the RAUC upgrade framework and the upcoming improvements.
He started introducing the upgrade bundle format in previous releases of RAUC and some problems with it: suboptimal performance in signature verification due to the need to read the entire bundle upfront, and possible malicious attacks to verification. Those cannot be fixed as they are due to the file format, and the security flaw could only be mitigated.
Then he described the entirely new bundle format that has been devised for future versions. It is based on dm-verity for image verification, which in turn uses the kernel device mapper. Those tools in turn required to reorganize the entire upgrade mechanism around accessing the image in 4 kB blocks.
This new format allows both a fast initial verification and verified random access, removing the need for the initial full-image verification. It also enables streaming upgrades, downloading only the needed blocks from the network instead of storing the entire bundle on the device. It can also reuse blocks from the previously installed image for those portions that have not been modified. The latest improvement though works only for block-aligned storage such as ext4, not on heavily optimized filesystems such as squashfs, but work is in progress to have file-based hashing.
The new bundle format also allows bundle encryption, not easily doable with the old format. This is based on dm-crypt and is block-based as well, allowing streaming upgrades even from encrypted bundles.
He ended listing some “simpler” improvements, including a installation history log and a more fine-grained state machine of the various steps involved in the upgrade process, including an interactive user confirmation step.
Talk “Booting Automotive ECUs really fast with modern security features” by Brendan Le Foll – BMW
Talk selected by Michael Opdenacker
This refreshing talk shares practical experience booting an embedded Linux system very fast but without sacrificing functionality. BMW’s Node0 In Vehicle Infotainment solution is supposed to be started within two seconds, in particular to show the rear camera within this time frame, but for lots of other reasons.
Using Suspend to RAM could seem like an attractive solution to achieve this, but it could drain the battery in a few days by keeping the system always on. Another reason is that not all SoC vendors support this feature in their kernel. Hibernation is interesting too but too damaging on automotive flash where a lifetime greater than 15 years is expected.
So, BMW engineers use a standard cold boot, and reach user space in about one second. They are able to achieve this without even sacrificing Secure Boot which hardly makes a difference according to them. Their system is also started by systemd, but they avoid starting Udev too early as it is too slow and noisy. Instead they pre-allocate block devices through an “early” target which also starts the early applications. They also use DM-verity to replace the kernel module signing and encrypted firmware checks.
Other tricks for booting faster are using the “Big” cores on Big/Little systems, booting with less RAM and hotplugging more later, keeping CPUs at their maximum frequency, using the first flash / eMMC sectors which are usually faster, starting critical applications with “near RT” priorities, replacing DBUS by “Smolkit” (to be shared soon), and starting the containerized system using LXC, whith is 8x faster than other container solutions.
A great practical boot time reduction talk showing that you can boot fast without sacrificing standard system features!
Talk “Beyond Complex Cameras: Complex Video Graphs Using PipeWire” by George Kiagiadakis
Talk selected by Miquèl Raynal
George started his talk presenting what a normal pipeline looks like. Problem is, modern cameras are not so simple anymore, it’s actually the norm today to have three or more sensors for a single shot on modern smartphones. The additional processing load must be shared across many different hardware and software blocks (Camera MCU, ISP, AI accelerators, CPU, etc) but the boundaries are not well defined yet. Not mentioning the fact that the pipelines are not linear either, the images going back and forth between all those processing blocks. Under Linux, the software responsible for orchestrate all of that is libcamera.
But what if we want to combine the images coming from different devices instead? In this case, after enumerating all the additional needs this requirement brings, George proposed to use PipeWire, which for instance allows to build very complex pipelines while still allowing processing to be done by separate userspace processes, on both video and audio. He actually compared PipeWire with a multimedia bus.
Eventually, George argued that in order to easily plumber everything together, he got involved in the development of a third tool, named WirePlumber, providing a scriptable interface to easily build processing graphs. He then proposed an interesting series of live demos showing how WirePlumber can help building more and more complex graphs, both through his scripts starting gstreamer instances and a GUI which clarifies what is being done and allows further plumbing as well. A nice and highly understandable approach to the world of complex multimedia pipelines.
Talk “Linux on RISC-V and the New OS-A Platform” by Drew Fustini
Talk selected by Paul Kocialkowski
As the RISC-V architecture is gaining wide adoption and interest from all sides of the embedded ecosystem, Drew Fustini gave his now-traditional talk about the current status of its support in popular free software projects. It starts with a number of clarifications about the different fashions of RISC-V (with various extensions and related profiles) and the intended use cases they cover. Actual implementations are also highlighted, both in the form of “open” cores (for which the hardware description language is available under a free software license) and physical chips and boards that are just starting to be produced in high quantities and available to the masses.
The latest enablement work done on the Linux kernel is then depicted, with a particular emphasis on the effort to support the Allwinner D1 MMU, with a good level of technical details about the associated challenges. Boot aspects are also mentioned, with details about the SBI interface and ongoing efforts towards ACPI/UEFI to match strongly-rooted expectations from server manufacturers. The status of support in typical distributions and build systems is also presented during the talk.
Finally the talk gives a nice highlight of the community focused on free software support for RISC-V, including information to get involved and participate (even without actual hardware in hands)!
Talk “Trading Fbdev for DRM, no returns accepted” by Geert Uytterhoeven
Talk selected by Théo Lebrun
The Fbdev subsystem came alive as the first platform-independent framework for graphical consoles on Linux. At the time, graphics was simple: a framebuffer lived somewhere in memory, being accessible from userspace through mmap(/dev/fbX). Mode enumeration and setting was done through ioctl() calls, and the subsystem exposed optional text-oriented hardware accelerations (monochrome-to-color expansion, rect fill/copy, pan/wrap). Userspace was responsible for exploiting more advanced hardware acceleration when needed.
A new framework, DRM, was created when hardware became more advanced; it focused on true-color graphics for desktop, having support for more and bigger pixel formats, multiple planes, 3D acceleration, etc. Text accelerations were less of a focus for this generation of hardware, and memory management needed handling by the subsystem as multiple planes were most-often involved.
DRM got Fbdev emulation through a shadow buffer exposed as /dev/fbX, which often got used to implement early text consoles. DRM being capable of emulating Fbdev, calls to deprecate it quickly appeared. Its deprecation was official in 2015, with the statement that no new Fbdev drivers would get merged. The second part of this talk compares both subsystems by converting an Fbdev driver to become a DRM one: ataridrm.
Issues were faced in this endeavour: lacking support for low-color pixel formats in DRM and issues with endianness (formats that are not 8-bit multiples, GUI toolkits that only generate native byte order which incurs software conversion after userspace rendering). The main still-standing issue is performance-related: all DRM drivers expose at least XRGB8888 which is extremely wasteful for monochrome-capable displays such as on the Atari. Some devices simply do not have the memory bandwidth necessary.
My takeaway from this is a clearer vision on the objectives of both DRM and Fbdev. One is aimed at high-end desktop graphics with full-color support and advanced hardware features while the other aims at exposing uncomplicated display controllers. The future is unclear, but it seems Geert Uytterhoeven argues that Fbdev has a spot to fill, especially in the embedded field. Hoping I haven’t spoiled the fun that can be found in this talk, I’d recommend you watch it even if it’s only as a really interesting history lesson.
Talk “Implementing UEFI-based Secure Boot + OTA Update for Embedded ARM Devices” by Jan Kiszka & Christian Storm
Talk selected by Thomas Petazzoni
Both speakers, Jan and Christian, work for Siemens, which is a strong contributor to the Linux Foundation CIP project. One of the thing that CIP tries to do is define best practices on how to use Linux for embedded applications. As part of this overall goal, Jan and Christian have in particular been working on secure boot and remote firmware updates: they are many solutions available, and it’s not always easy to find the “right” one.
Jan and Christian started by explaining that on Intel x86/x86-64 systems, CIP already has a well-defined secure boot strategy: based on UEFI, a minimal UEFI bootloader called efibootguard, the Unified Linux Kernel Image mechanism, and dm-verity and/or dm-crypt. The advantage is that the solution needs to be implemented only once, and works on all x86/x86-64 platforms that are UEFI compliant.
However, as they described, on ARM/ARM64, the situation is more complex, with custom engineering being needed for each platform. Their proposal is to standardize on UEFI as well, using either EDK2 or U-Boot as UEFI providers (even though their preference clearly leaned towards U-Boot). This allows to re-use efibootguard, to use the Unified Linux Kernel Image, and have a standardized booting process that allows to leverage all what is used on x86.
In terms of software update, the CIP project has settled on using swupdate. They’ve discussed how they match an updated kernel with its root filesystem using a new IMAGE_UUID parameter in the /etc/os-release file, and also described how dm-verity is used for the verification of the root filesystem.
Overall, even though maybe not everyone will want to use this solution as-is, it was a very informative talk and certainly provides interesting ideas.
thx. great collection.