After a first and a second episode, our series of blog posts with our selection of talks we liked at the latest Embedded Linux Conference continues. Read on to discover the last 3 talks that we enjoyed and decided to summarize and highlight for you.
Rethinking U-Boot Devicetree Story – Sumit Garg, Linaro
Talk chosen by Bootlin engineer Thomas Petazzoni
In this talk, Sumit Garg, an engineer from Linaro, tried to draw a future for how Device Tree files could be managed. Indeed currently, the Linux kernel repository serves as a de-facto reference location for Device Tree source files (but also Device Tree bindings), and then those are more or less replicated in other projects, including U-Boot, but also TF-A or OP-TEE, even though Sumit really focused on U-Boot in his discussion. This leads to a synchronization nightmare.
From Sumit’s perspective, the Device Tree should be provided by the firmware to the OS, and therefore when U-Boot is the firmware/bootloader, the Device Tree should be provided by U-Boot to the Linux kernel, rather than the Linux kernel having its own copy. Turns out that the devicetree-rebasing Git repository already provides an operating system agnostic place with all DT bindings and DT files from the Linux kernel, regularly updated. This repository is now synchronized regularly within dts/upstream/ in U-Boot, and the new CONFIG_OF_UPSTREAM option in U-Boot allows to use DT files from dts/upstream/. See this commit for some details. Thanks to this, the DT files are no longer copied/modified for U-Boot specifically, and also the tooling for DT bindings validation can be used as expected.
This talk was interesting because it highlighted some real work that is already on-going, and Sumit tries to push forward a problem that has remained unresolved for quite some time, even though it annoys a lot of developers working on Linux/U-Boot.
A fully open source stack for MIPI cameras, by Bryan O’Donoghue
Talk chosen by Bootlin engineer Jérémie Dautheribes
Many Linux users, like myself, might have had the unpleasant surprise of discovering that their new laptop features an IPU6 Intel webcam that is not yet supported by the mainline kernel and relies heavily on closed-source firmware. Thankfully, some people have been tackling this problem for a few years now.
In this talk, Bryan O’Donoghue from Linaro presents the work he has been doing together with Hans de Goede from Red Hat to make a fully open-source stack for MIPI cameras, such as the Intel IPU6.
In the first part, he introduces the basics of what a camera does, explaining especially the Bayer encoding/decoding mechanism, which allows a digital camera to capture light and colors. But capturing light and colors is not enough. There are other important parameters to handle like the 3As (autofocus, auto white balance, and auto exposure/gain), along with more advanced features like auto contrast and skin tone adjustments.
All of this is usually controlled and computed by an Image Signal Processor (ISP), which consists of a series of hardware blocks sometimes running firmware. While all of this has historically been hidden behind USB cameras, which are well-supported by Linux, more and more webcams are now MIPI cameras delivering raw Bayer data directly to userspace. This has the advantage of allowing more control over the processing and the various advanced features. However, vendors tend not to disclose their “secret sauce” algorithms, as Bryan says, and it can be hard to have full control over it.
That’s why Linaro and Red Hat ended up making a generic open-source implementation that solves this problem across platforms. In the second part of the talk, Bryan presents the efforts made to handle this topic. The idea is to have an open-source software reimplementation of what happens in the ISP and various closed firmwares. For this purpose, they use libcamera, which is the de facto OSS standard. It handles and hides all the complexity of pipeline configuration and processing. But applications need a way to access the camera through libcamera: nowadays, the easiest way is to use PipeWire, which is becoming the standard media server on Linux. For full integration, Jan Grulich from Red Hat has added support for PipeWire cameras in Firefox (this needs to be manually enabled through about:config though).
In the final part, Bryan concludes by outlining future plans. The main goal is to enhance image quality by implementing or improving algorithms for digital image processing. As all of this processing can require a lot of CPU computing, especially on embedded targets, they are also working on using GPU acceleration through OpenGL or OpenCL/Vulkan.
At the end, we even had a demo of this fully open-source stack! While the image quality is not perfect — yet 😉 —, it is IMHO enough for basic use cases. I’m looking forward to seeing the evolution of this project and I can’t wait to use it on my own laptop!
Standardizing the Generation and Signing of Boot Images, by Neha Francis & Vignesh Raghavendra, Texas Instruments
Talk chosen by Bootlin engineer Köry Maincent
In this talk, Vignesh Raghavendra and Neha Malcom Francis, two engineers from Texas Instruments (TI), discuss how SoCs can leverage Binman to simplify the overall build flow. Simon Glass, an engineer from Google, significantly contributed to mainlining the K3 architecture but was not present.
Vignesh Raghavendra begins the presentation by explaining the necessity of Binman. In traditional embedded systems, only the bootloader and environment were needed to boot a board and run Linux, allowing for the binaries to be appended one after another with the ‘cat’ command. With the introduction of SPL, it became possible to place part of the code in tiny memory.
Nowadays, the ARM64 boot flow uses a combination of ROM code, SPL, S-EL1 firmware, EL3 firmware, and U-Boot. Managing multiple firmware for heterogeneous SoCs with multiple MCUs and MPUs requires more sophisticated tools. The simple ‘cat’ command is no longer sufficient for generating the complete boot firmware binaries, nor can it sign or encrypt images. This is where the Binman tool becomes essential.
Neha Malcom Francis then takes over to explain how Binman works and how they used it to generate a firmware image for the TI K3 SoC. Binman uses a simple device tree-like image description to generate firmware and is capable of managing image encryption and signing.
Finally, she outlines the advantages of using Binman and discusses future developments in the pipeline. We found this talk quite enlightening on how Binman is used to allow the creation of complex binary images needed for modern SoCs.
