Snagboot 1.3 release overview

Snagboot, Bootlin’s vendor-agnostic recovery tool for embedded platforms, has been under continued development since its first release in May 2023. Let’s take a look at the main changes brought by version 1.3, published just a few days ago.

Passing USB device paths

Both snagrecover and snagflash previously relied on USB vendor ID:product ID addresses to find recovery devices exposed by boards. This was problematic in the case where two boards with the same vendor and product IDs had to be recovered or reflashed simultaneously. To solve this issue, we added the ability for snagrecover and snagflash to use USB paths of the form bus:port1-port2-... instead.

The main challenge we faced while implementing this feature was integration into the NXP i.MX recovery process. In Snagboot 1.2, i.MX recovery relied on the Python ctypes bindings to the hidapi library. These bindings did not allow searching for an HID device using bus/port paths. This, and other issues we encountered with hidapi bindings led us to drop them entirely and reimplement a partial HID library inside the Snagboot codebase.

i.MX53 support

Martin Fuzzey contributed a set of changes to snagrecover that added support for the i.MX53 family of SoCs. A few changes to the protocol layer of Snagboot were necessary since these SoCs used a raw USB device instead of an HID device.

Improvements to the AM335x setup script

Recovering AM335x platforms with Snagboot requires setting up a recovery shell using a special helper script. We’ve cleaned up the syntax and some of the logic of this script to bring it closer to POSIX compatibility and allow non-Bash shells such as Fish shell to run it successfully.

These are the main changes brought by v1.3. If you’re curious to see more, you can view the full changelog here. Thank you to all the people who contributed to this release!

Bootlin at Embedded Open Source Summit North America, Apr 16-18: 4 talks, 11 engineers

Embedded Open Source SummitThe Embedded Open Source Summit will take place on April 16-18 in Seattle (Washington, US), with a wide range of events, including the Embedded Linux Conference which is of particular interest to Bootlin considering our core expertise.

As usual, a significant fraction of the Bootlin team will attend the event, with no less than 11 engineers from our team going to Seattle: Alexandre Belloni, Bastien Curutchet, Grégory Clement, Jérémie Dautheribes, João Marcos Costa, Köry Maincent, Louis Chauvet, Luca Ceresoli, Maxime Chevallier, Thomas Petazzoni and an additional engineer who will join us at the beginning of April. At Bootlin, all engineers, regardless of their seniority level, are offered the possibility of attending up to 2 conferences per year, to expand their knowledge on embedded Linux topics, to create connections with the community and therefore provide better services to our customers.

In addition to participating, we will also be delivering no less than 4 talks at the conference:

  • In the Kernel Trenches: Mastering Ethernet Drivers on Linux – Maxime Chevallier, Bootlin
    An Ethernet Controller driver in linux is a complex piece of software, with many entrypoints and callbacks, even for a basic Ethernet Controller we can find in Embedded Systems. In this talk, we’ll take a tour of what’s inside a typical Ethernet driver. We’ll start with the base of what every network driver provides, even non-Ethernet ones, a net_device. We will discuss the various required steps to get a basic driver working, ranging from simple initialization, queue setup, DMA mappings, before diving into more advances topics such as the NAPI loop implementation, memory-management through the page-pool API and finally XDP, which is now getting implemented in more and more drivers. We’ll continue the tour through the ethernet-specific bits, such as the ethtool operations, and phylink support to interact with a potential Ethernet PHY, and discuss a little bit about the specifics of embedded Ethernet Controllers, where low-level configuration for a PCS or some Serdes lanes might be required. Finally, we’ll discuss about testing and debugging tools that can be useful to help you in your driver development, but also in debugging and optimizing existing drivers.
  • Introduction to DAPM, Linux Power Management for Embedded Audio Devices – Luca Ceresoli, Bootlin
    DAPM (Digital Audio Power Management) is the part of ASoC (Alsa for System-on-Chips) that dynamically turns on and off power to various parts of the audio hardware in embedded Linux systems. DAPM is very useful, but not well known and documented. After introducing the motivations for DAPM and how it models the hardware, Luca will present some user space tools to understand how DAPM works at runtime. He will then show what DAPM practically means for developers, with code examples of how to add DAPM support to device drivers. Finally he will mention some improvements to the DAPM tooling and documentation.
  • Unlocking the Potential of Suspend to RAM Using Linux in a Multi-Core, Multi-Firmware Automotive SoC – Grégory Clement, Bootlin
    Fast booting in modern, complex System-on-Chips (SoCs) demands innovative solutions. This talk delves into the intricacies of implementing Suspend to RAM (S2R) on a sophisticated SoC featuring diverse CPU cores—high-performance A72, real-time R5, and low-power M3—as well as multiple firmware components, including open-source ones like TF-A, U-Boot, and OP-TEE. We begin with a hardware overview of the TI DRA821 SoC from the Jacinto family, emphasizing the open-source firmware utilized in the boot process. The focus then shifts to presenting a practical approach for integrating S2R into this multifaceted platform. Notably, our implementation poses a unique challenge: the SoC is entirely powered off during S2R, adding an extra layer of complexity. Join us as we share insights into the hurdles encountered and resolved during the implementation, along with a discussion on the outstanding challenges for which we’ve devised viable solutions. This session offers a comprehensive exploration into the software, including bootloaders and the Linux kernel, as well as the hardware aspects, showcasing a real-world approach to bringing Suspend to RAM to life on a cutting-edge automotive SoC.
  • Adding Support for Power Over Ethernet (PoE or IEEE Clause 33) to Linux Network Stack – Köry Maincent, Bootlin
    Power over Ethernet (PoE) is a technology that combines electrical power and data transmission over a single Ethernet cable. It eliminates the need for separate power sources, simplifying installations for devices like IP cameras, and VoIP phones. In this presentation, we’ll initially delve into Power over Ethernet (PoE), It debuted in IEEE Clause 33 without explicit reference to the PoE nomenclature. We will look at what currently exists in the Linux kernel and user space to support PoE. We’ll continue our discussion with some details of the Linux implementation currently in development and the PSE core changes brought by this new support. The PSE framework core and bindings happen to need modification as it was not prepared for the PoE specificities. In parallel to these extensions of the PSE framework, we developed Linux kernel drivers for two distinct PoE controllers: the Microchip PD692x0 and the Texas Instruments TPS23881. Finally, we will look into the mainline status, the things that still need to be merged and the future features that need development. This PoE Linux support is sponsored and funded by DENT Project.

See the full schedule of talks.

The Embedded Open Source Summit isn’t just about the presentations—it’s about forging connections, sharing ideas, and exploring opportunities. Join us for insightful conversations with the Bootlin team, where you can discuss your projects, ambitions, and maybe even discover new career pathways.

And mark your calendars because following the summit, our CEO Thomas Petazzoni will be in the Bay Area until April 24, ready to delve deeper into your embedded Linux needs.

So, whether you’re a seasoned veteran or a curious newcomer, don’t miss out on the Embedded Open Source Summit 2024. We’ll see you there!

Linux 6.7 released, Bootlin contributions

The Linux 6.7 kernel was released almost two weeks ago, with as usual plenty of new features and updates, better described by the excellent LWN.net: part 1, part 2. On our side, while we continue to submit a significant number of patches, this release has been somewhat more quiet, with only 27 patches integrated.

Continue reading “Linux 6.7 released, Bootlin contributions”

Welcome to João Marcos Costa and Bastien Curutchet

Welcome on board!We are very happy to welcome to our team João Marcos Costa and Bastien Curutchet.

João joined our team at our Lyon office at the end of November 2023. After a double degree from ENSICAEN in France and Universidade Federal do Rio Grande do Norte in Brazil, João started his professionnal career in Brazil, and then moved to France where he worked for Witekio and Alstom. Thanks to his past experiences, João brings relevant expertise in embedded Linux, and especially Yocto integration. And for João, joining Bootlin is actually come back, as he did one of his internships at Bootlin in 2020, during which he wrote and contributed to upstream U-Boot the support for the SquashFS filesystem.

Bastien joined our team at our Toulouse office at the beginning of January 2024. After graduating from INSA Lyon in 2016, Bastien spent a short time at Texas Instruments and then moved to Scalian, where he has been involved in numerous embedded Linux and Linux kernel projects, but also in VHDL development for FPGAs. Bastien is well-versed in build system management, having overseen the development of custom Linux firmwares based on Yocto, applied to platforms like iMX6 or Zynq Ultrascale+. Bastien will be helping our team handle more Linux kernel and low-level development projects for our customers.

Welcome João and Bastien!

Bootlin toolchains 2023.11 released

Bootlin toolchains 2023.11Our toolchains.bootlin.com project offers a wide range of freely available pre-compiled cross-compilation toolchains, updated on a regular basis.

We are happy to announce the availability of version 2023.11 of our toolchains. Even though they are released in December (so one would expect the version number to be 2023.12), they are all generated using Buildroot 2023.11, hence the release number used for the toolchains.

Compared to the previous 2023.08 toolchains, here are the relevant changes:

  • Bleeding edge toolchains are now using gdb 14.1 instead of 13.2, glibc 2.38 instead of 2.37 and uclibc 1.0.45 instead of 1.0.43. They continue to use gcc 13.2 and binutils 2.41, as well as musl 1.2.4
  • Stable toolchains are now using gdb 13.2 instead of 12.1, glibc 2.38 instead of 2.37 and uclibc 1.0.45 instead of 1.0.43. They continue to use gcc 12.3 and binutils 2.40, as well as musl 1.2.4
  • The glibc version used has the fix for CVE-2023-4911
  • The gdb build has been fixed to no longer rely on uninstalled libbfd.so and libopcodes.so libraries
  • The zlib library, which was incorrectly present in the toolchain sysroot, is gone, fixing various build failures encountered with 2023.08 toolchains
  • There are now toolchains for m68k 68xxx based on uclibc and musl in addition to glibc, which was already supported

We have already submitted a patch to update the support of those toolchains in Buildroot, where they can be used as external toolchains.

Hopefully these toolchains will continue to be useful to the embedded Linux community. For any question, feedback or issue, you can use the Github issue tracker of the project.

Bootlin at Open Source Experience and SIDO in Paris, Dec 6-7

Paris will be hosting next week-end a combined event composed of the Open Source Experience and SIDO, the first dedicated to open-source technologies, and the second to IoT, AI, digital infrastructure and cybersecurity.

Open Source Experience

Thomas Petazzoni, Bootlin CEO, will be representing Bootlin at these events, and will also be participating to the round table Embedded systems security: a technical and organizational approach on December 7, at 2:30 PM UTC+1. The abstract of the round table is:

Security is a major issue. Embedded systems are increasingly complex and connected, making them more vulnerable. The aim of this round table is to discuss best practices for guaranteeing security

Thomas will be speaking with Daniel Fages (Freelance), Eloi Bail (Savoir Faire Linux) and Jean-Charles Verdié (Canonical), and the round table will be moderated by Cédric Ravalec (Smile).

If you’re interested in discussing career, business or partnership opportunities with Bootlin, do not hesitate to contact Thomas Petazzoni ahead of the event to schedule a meeting.

Bootlin establishes a strategic partnership with Ratiotech, an expert in electronics and hardware design

For close to 20 years, Bootlin has offered to companies around the world its expertise in embedded Linux system development, with engineering and training services covering low-level software development in the Linux kernel and open-source bootloaders, as well as embedded Linux system integration and build systems.

Bootlin’s expertise is obviously tightly coupled with hardware engineering, as the low-level software we develop runs on a wide variety of hardware platforms, sometimes with complex designs. As experts in low-level software, our team possesses a robust understanding of the hardware used in embedded systems, including SoCs, board design, hardware interfaces, and protocols.

Today, we are delighted to announce a strategic partnership with Ratiotech, marking a significant expansion of our collective expertise. Ratiotech brings an impressive 25 years of experience in hardware design, electronic systems, signal integrity, and EMC.

Bootlin / Ratiotech partnership

This partnership enables:

  • Bootlin to offer significantly deeper and broader expertise in investigating, debugging, and resolving issues during the bring-up of our customers’ hardware platforms
  • Ratiotech to leverage Bootlin’s solid expertise in embedded Linux development, providing customers with a comprehensive range of services covering both hardware development and Linux operating system integration
  • A collaboration that allows Bootlin and Ratiotech to provide turn-key solutions, encompassing the entire development of hardware/software solutions: hardware design, industrialization, low-level software development on microcontrollers, porting and integration of embedded Linux systems, and application development

Commencing in early 2024, Bootlin and Ratiotech will share offices near Toulouse, France, facilitating a seamless collaboration between the teams of both companies to ensure the success of this partnership.

Ratiotech’s CEO, Fabien Hue, states, “This strategic partnership is an excellent step to accelerate the development of Ratiotech and offer our customers a broader expertise, fully complementing Bootlin’s expertise in embedded Linux systems.”

Thomas Petazzoni, Bootlin’s CEO, adds, “The expertise and offerings provided by Bootlin to its customers will be significantly enhanced through this partnership with Ratiotech. This close relationship will not only expedite the bring-up of new hardware platforms but also enable us to offer essential hardware development, debugging, and investigation expertise that many of our customers require. We are eagerly looking forward to further assisting our customers with Ratiotech’s expertise.”

For more details on this partnership, check our hardware expertise, and for more details about Ratiotech, visit their website at Ratiotech.

Keep your Yocto layer simple! Introducing meta-kiss, a working reference Yocto/OE setup

At Bootlin we help many of our customers using Yocto/OpenEmbedded to build the Linux software stack running on their end products. While doing that we have seen all sorts of problems caused by all sorts of complicated code in their build system setup. So we wondered what we could do to improve the overall situation. Continue reading “Keep your Yocto layer simple! Introducing meta-kiss, a working reference Yocto/OE setup”

Open-source Linux kernel support for the Allwinner V3/V3s/S3 H.264 video encoder

Bootlin has been involved with improving multimedia support on Allwinner platforms in the upstream Linux kernel for many years. This includes notable contributions such as hardware-accelerated video decoding initial support for MPEG-2, H.264 and H.265 following the successful crowd-funding campaign in 2018, support for the MIPI CSI-2 camera interface and early support for the Allwinner V3/V3s/S3 Image Signal Processor (ISP) in mainline Linux.

In addition to this work focused on Allwinner platform, we have also developed and released in 2021 initial Linux kernel support for the Hantro H1 H.264 stateless video encoder, used on Rockchip processors, on top of the mainline verisilicon/hantro driver.

The Allwinner V3s chip on the Lichee Pi Zero

Today Bootlin is happy to announce the release of Linux kernel support for H.264 video encoding with the Allwinner V3/V3s/S3 platforms, in the form of a series of patches on top of the mainline Linux cedrus driver (which already supports decoding) and a dedicated userspace test tool. The code is available in the following repositories:

The updated cedrus media controller graph.

This work is both the result of our internal research and a continuation of earlier projects that were carried out by members of the linux-sunxi community to document and implement early proofs of concepts for H.264 encoding, covering older platforms such as the A20 and H3. We would like to give a warm thank-you for these previous contributions that made life significantly easier for us.

Adding support for H.264 encoding required bringing a significant architecture rework to the driver, which was rather messy and not very well structured (but it was of course quite difficult for us to find the most elegant way of writing the driver back in 2018, when we were just getting start on the topic). With a clear organization and the adequate abstraction in place, it became much easier to add encoding support and focus on the hardware-specific aspects.

Just like decoding, encoding is based on the V4L2 M2M framework and shares the same V4L2 and M2M devices with decoding (meaning that only a single job of either decode or encode can be processed at a time). Two video device nodes are exposed to userspace, allowing as many decoding and encoding contexts as needed to exist simultaneously. Advanced H.264 codec features such as CABAC entropy coding and P frames are supported.

However this work is not yet suitable for inclusion in the mainline Linux kernel since there is no well-defined userspace API (uAPI) for exposing stateless encoders. We have started discussions to converge towards an agreeable proposal that would be both generic enough to avoid device-specific considerations in userspace while also benefiting from all the features and specificities that stateless encoders can provide. While discussions are still in progress, others have expressed interest in the same topic for the VP8 codec. Our previous work published in 2021 for the Hantro H1 H.264 stateless video encoder is also not upstream yet for the same reason: the need for a well-defined userspace API to expose stateless encoders.

Besides the introduction of a relevant new uAPI, a number of challenges remain in the path towards full support for Allwinner H.264 video encoding in mainline Linux:

  • The rework of the driver needs to be submitted and merged upstream;
  • Rate-control is currently not implemented and only direct QP controls are available;
  • This work only covers the Allwinner V3/V3s/S3 platforms, while most other generations also feature different (yet rather similar) H.264 encoder units that could also be supported with some dedicated effort;
  • Pre-processing features such as scaling and pixel format conversion are not yet support;
  • Developing userspace library support (such as FFmpeg or GStreamer) to make use of the stateless encoder uAPI would be necessary;

If you are interested in funding the effort to help us advance any of these topics, feel free to get in touch with us to start the discussion!

Back from Netdev 0x17

At Bootlin, we focus on Embedded Linux development and support, and these embedded devices often have a network interface, be-it an Ethernet port, a Wireless chip or some other kind of communication channel that falls under the Linux Networking Stack’s framework.

So it’s always interesting to see what the rest of the community is working on, and meet in real life people we interact with on the netdev mailing list.

That’s why this year, Alexis Lothoré and Maxime Chevallier flew to Vancouver to participate to the Netdev Conference, a 5 days event organised by the Netdev Society, a small non-profit run by volunteers dedicated to holding this event.

Bootlin at Netdev

Most talks at Netdev are not directly covering topics we’re actively working on, but it’s always refreshing to see these new exciting technologies that could trickle their way down to the embedded world a few years from now. It is also always pretty interesting to stay up to date about challenges encountered by other parts of the networking industry, at scales way different than the ones we are used to.

We learned for example what CXL is about, what it brings and the effort that are made to design new networking hardware around this technology to change the way we think about datacenter networking.

When we attended Netdev 0x13 in 2019, QUIC was one of the hot topics. This year, Homa was under the spotlight with talks on what it is, and how this new protocol could address some of TCP’s problems.

Like all previous editions, we learned all the progress that were made with TC and its future, new ways of bypassing the kernel stack, BPF integration in the kernel, along with XDP which continued to be more and more powerful.

Another hot topic in the kernel is the introduction of the Rust language, and the network subsystem is a pretty relevant target for the new features brought by the language. As a consequence, Rust subsystem maintainers Miguel Ojeda and Wedson Almeida Filho gave an overview of Rust benefits compared to traditional C code, and then showed a step-by-step implementation of a kernel-side TCP server module. While this example is not perfectly representative of classic network-related drivers we usually write, it was a nice showcase of current state of kernel APIs abstractions in Rust.

We also discovered the new use-case that is now driving most of the datacenter networking efforts, which is without surprise AI and Machine Learning. Turns out, if you want your ChatGPT to answer up-to-date replies without having to wait for too long, you need a powerful and well-organized datacenter for the training part, and networking engineering takes a big part in it to keep all those GPUs fed at a relevant pace.

This lead to the devmem TCP effort, which started to feel a bit familiar for us as it uses dma-buf, which we also sometimes use on multimedia pipelines. The ML and AI topic was introduced to us by the wonderful Keynote session given by Manya Ghobadi, who got all the audience captivated by how AI and ML works, what AI workloads requires in terms of network traffic scheduling, datacenter topology and computing hardware that uses optical computing.

On the final day, we even had a visit from Jakub Kicinski (one of the co-maintainers of Linux networking tree), presenting what he had been working on, and gave us an update on the netdev development statistics (and basically, his main point is that we do need to review more patches).

For the first time, there was a talk from Bootlin at netdev, as Maxime presented one of the topics he’s been working on lately : Improving multi-PHY and multi-port interfaces support. Although it was one of the only talks focusing on the low-levels aspects of the Ethernet stack, it triggered some discussions and interest from the community, which will help further improving the ongoing work.

The slides and videos of the event will be published at some point in the future, we will for sure mention this to our readers when it becomes available.

We’ll conclude this short feedback by thanking once again the Netdev Board members, organizers, speakers and the audience for this great event.

We’ll come back 🙂