Embedded Linux Conference Europe 2022: four talks from Bootlin

The schedule for the upcoming Embedded Linux Conference Europe 2022 has been published recently.

Bootlin CEO Thomas Petazzoni is again a member of the program committee for this edition of ELCE, and has helped with other members of this committee in reviewing and selecting the numerous talk submissions that have been received.

Bootlin will obviously be present at this conference. With 13 engineers from Bootlin participating, almost the entire company will be in Dublin for this major event of the embedded Linux community. Also, 4 of the talks that we had submitted have been accepted:

  • Luca Ceresoli on Basics of I2C on Linux
    This talk is an introduction to using I²C on embedded Linux devices. I²C (or I2C) is a simple but flexible electronic bus to allow low-speed communication between the CPU and all sorts of chips: PMICs, ADC/DACs, GPIO expanders, video sensors, audio codecs, EEPROMS, RTCs and many more. It is so popular that knowing it is a must for any embedded system engineer. Luca will first give an introduction to what I2C is at the electrical level. He will then describe how I2C is implemented in the Linux kernel driver model and how that appears in sysfs, how to describe I2C devices using device tree and how to write a driver for an I2C device. Finally he will present the tools to communicate with the chips from userspace and share some debugging techniques, including inspection of the physical bus and software-level debugging.
  • Miquèl Raynal on Improving Wireless PAN Support
    Anybody eager to learn about IoT devices has at least once tried to play with Zigbee or 6lowpan sensors. These two protocols are built on top of a well common MAC/PHY specification: IEEE 802.15.4, also known as Wireless Personal Area Networks: WPAN, designed to be low-rate/low-range wireless networks. There is already substantial support for this protocol in the Linux kernel but when my journey started, several of the MAC-related operations well described in the specification were not implemented, making the subsystem mainly useful for very simple use cases: peer-to-peer transmissions. This is unfortunate as a significant part of the idea behind WPAN is to make these networks quite adaptive and resilient, which requires a minimal subset of the peer management procedure to be supported. Besides a number of preparation changes, the main idea behind the continuous flow of patches was to bring support for the scanning procedure which allows a PAN controller to detect all the compatible devices around it in different ways. Discovering these devices is the first step in order to associate them together and build up starred networks. This talk will be an opportunity to explain the new APIs allowing such discoveries and provide a state of the art of the support in the mainline kernel.
  • Michael Opdenacker on Implementing A/B System Updates with U-Boot
    A popular way to implement system updates is through the A/B scheme, in which you have two copies of the root filesystem, one which is active, and one that is meant to contain the next update. When a new update is successfully applied, you need to make the corresponding partition become the new active one. That’s when a number of practical questions arise, such as how to identify the active partition, how to detect when the new system fails to boot properly, and how to fall back to the previous version? It was hard to find documentation about how U-Boot could address such needs to implement a functional and failsafe A/B system update mechanism. This presentation proposes to address this need by sharing the practical solutions we found, using lesser known commands and capabilities in U-Boot. We will also explain how the Linux side can cooperate with the U-Boot side. Fortunately, you won’t need to erase half of your brain to get updated on this topic.
  • Paul Kocialkowski on Walking Through the Linux-Based Graphics Stack
    The graphics stack used with the Linux kernel is a notoriously complex beast. From userspace down to the kernel level, a number of components are involved and interact with eachother. It is also an area that is constantly evolving to meet new use cases, refresh legacy implementations and achieve better performance. This makes it difficult to have a clear idea of the big picture and what is actually happening when using graphics-related components. This presentation will detail a walk through the graphics stack, with actual examples of displaying a buffer and rendering using the GPU. Going from the application level through the system libraries, down to the kernel and ending with actual hardware configuration. State-of-the-art technologies such as Wayland and DRM will be highlighted with relevant excerpts from the source code of related free software projects that are widely used today.

We look forward to meeting again the embedded Linux community, its developers, users and maintainers, at Dublin during this conference!

Bootlin at the Embedded Linux Conference Europe 2020

Embedded Linux Conference Europe 2020The schedule for the next Embedded Linux Conference Europe has been recently published, and Bootlin will once again be strongly present at this (virtual) event by giving a number of presentations. The registration for ELC-E is open, and due to the virtual nature of the event, the registration cost is only $50, which makes is accessible to pretty much everybody.

  • From the Camera Sensor to the User, the Journey of a Video Frame. In this talk, Bootlin engineer Maxime Chevallier will share his experience working in the Video4Linux subsystem, implementing support for the Rockchip camera interface controller, and a PAL/NTSC decoder used as the input source. Talk on Monday October 26 at 18:30 GMT.
  • Yocto Project and OpenEmbedded: A Collection of Best Practices. In this talk, Bootlin engineer, and Yocto Project expert and trainer Alexandre Belloni will share his experience of using Yocto Project and OpenEmbedded through a collection of best practices. There are indeed numerous ways of using OpenEmbedded and the Yocto Project, but some ways and solutions are better than others! Talk on Tuesday October 27 at 13:00 GMT.
  • Building Embedded Debian and Ubuntu Systems with ELBE. In this talk, Köry Maincent will share his experience using the ELBE build system, which can be used to automate the process of creating embedded Linux systems based on Debian or Ubuntu. Köry has contributed to ELBE the support for building Ubuntu systems, and has used ELBE on two different projects. This is an interesting alternative to the traditional cross-compilation approach taken by Yocto Project, OpenEmbedded or Buildroot. Talk on Tuesday October 27 at 15:15 GMT.
  • Using Visual Studio Code for Embedded Development. In this talk, Michael Opdenacker will share his experiments of using VS Code for embedded Linux development, which extensions are available to help navigate in the code, build and debug code, from kernel-space and user-space. Talk on Tuesday October 27 at 17:15 GMT.
  • Precision Time Protocol (PTP) and Packet Timestamping in Linux. Antoine Ténart has implemented PTP support in both an Ethernet switch driver, and an Ethernet PHY driver, both in the upstream Linux kernel. He will share his experience about PTP, its support in Linux, and its offloading at the MAC and PHY level. Talk on Tuesday October 27 at 19:30 GMT.
  • Supporting Hardware-Accelerated Video Encoding with Mainline. After working on the HW-accelerated video decoding on Allwinner platforms as part of our crowdfunded effort, Paul Kocialkowski recently worked on HW-accelerated video encoding on Rockchip platforms. In this talk, he will share the issues encountered, and what needs to be resolved to create a useful kernel to userspace interface to properly support stateless video encoders. Talk on Wednesday October 28 at 16:15 GMT.
  • Understand ECC Support for NAND Flash Devices in Linux. Miquèl Raynal, the Linux kernel NAND subsystem maintainer, has recently worked on improving support for various strategies to handle ECC for NAND flash devices. He will share some background information on ECC, why they are needed, how and where ECC are typically handled, and how the Linux kernel deals with the different possibilities. Talk on Wednesday October 28 at 18:30 GMT.

In addition to contributing talks, Bootlin CTO Thomas Petazzoni is also a member of the ELC-E program committee: he reviewed, ranked all talk submitted for the conference and participated with the rest of the committee to the selection of the talks that are now scheduled for the event.

Even though we once again won’t have the chance to meet our fellow members of the embedded Linux community in person, we look forward to attending a set of great talks, and have interesting discussions during the Q&A and through the instant messaging platform that will be available around the conference.

Back from ELCE 2019: our talks videos, slides, and more!

With 8 engineers participating to the Embedded Linux Conference Europe, almost the entire Bootlin engineering team took part to the conference. As usual, we not only attended the event, but also contributed by giving a total of 5 talks and 2 tutorials, for which we’re happy to share below the videos and slides. Also, as part of this conference, Bootlin CTO Thomas Petazzoni received an award for his contribution to the conference.

Buildroot, what’s new ?

Talk given by Thomas Petazzoni, slides in PDF and slides source code.

Timing boot time reduction techniques

Talk given by Michael Opdenacker, slides in PDF, slides source code.

Integrating hardware-accelerated video decoding with the display stack

Talk given by Paul Kocialkowski, slides in PDF, slides source code.

RTC subsystem, recent changes and where it is heading

Talk given by Alexandre Belloni, slides in PDF, slides source code.

Flash subsystems status update

Talk given by Miquèl Raynal (from Bootlin) and Richard Weinberger (from sigma star gmbh), slides in PDF, slides source code.

Offloading network traffic classification to hardware

Talk given by Maxime Chevallier, slides in PDF, slides source code.

Introduction to Linux Kernel driver programming

Tutorial given by Michael Opdenacker, slides in PDF, slides source document. The video is not yet available, but should be published in the future.

Introduction to the Buildroot embedded Linux build system

Tutorial given by Thomas Petazzoni, slides in PDF, slides source code. The video is not yet available, but should be published in the future.

Award to Thomas Petazzoni

During the traditional closing game of the conference, we were really happy to have Bootlin’s CTO Thomas Petazzoni called on stage, to receive from the hands of Tim Bird, an award for his continuous 11 year participation to the conference, with 24 presentations given, one keynote and for the past two years, participation to the conference program committee. We are honored and proud by this recognition of Thomas contribution to the conference.

Thomas Petazzoni receives ELCE conference award

Thomas Petazzoni receives ELCE conference award

Thomas Petazzoni receives ELCE conference award

Bootlin at the Embedded Linux Conference Europe 2019

This year, Bootlin missed the Embedded Linux Conference North America which took place late August in San Diego, US. It was the first time in many years that Bootlin was completely absent from an Embedded Linux Conference.

But the coming Embedded Linux Conference Europe is going to be different in that respect: Bootlin will once again have a strong presence at this event, which in 2019 takes in Bootlin’s home country, France, from October 28 to October 30. And this year, ELCE is not only in France, but more precisely in Lyon, the city where one of the 3 Bootlin offices is located, so for some of our engineers it will be a very local conference!

Embedded Linux Conference Europe 2019

Of Bootlin staff, Michael Opdenacker, Alexandre Belloni, Grégory Clement, Kamel Bouhara, Miquèl Raynal, Maxime Chevallier, Paul Kocialkowski and Thomas Petazzoni will participate to the event. The schedule of the event, which also includes talks from the Open Source Summit and a number of co-located events, shows a nice selection of interesting talks.

As usual, we don’t limit our participation to just attending: we also give talks and tutorials. This year, the following proposals we made have been accepted:

In addition to being present at the Embedded Linux Conference Europe, Bootlin will also be present:

  • at the Buildroot Developers Meeting, which will take place on the 3 days before the conference, through to the participation of Buildroot co-maintainer Thomas Petazzoni
  • at the Linux media summit, which will take place in parallel to the Embedded Linux Conference Europe. Bootlin engineer Paul Kocialkowski will be part of the attendees.

Back from ELCE 2018: our selection of talks

The Embedded Linux Conference Europe edition 2018 took place a few weeks ago in Edinburgh, Scotland, and no less than 9 engineers from Bootlin attended the conference. While our previous blog post shared the videos and slides of our talks, tutorials and demos, in this blog post we would like to highlight a selection of talks that Bootlin engineers found interesting. We asked each of the 9 engineers who attended the event to pick one talk they liked, and make a small write-up about it. Of course, many other talks were interesting and what makes a talk interesting is very subjective!

Getting Your Patches in Mainline Linux: What Not To Do (and a Few Things You Could Try Instead), by Marc Zyngier

Talk selected by Maxime Ripard

Marc gave a talk on a subject that is often debated, and still confusing to newcomers: how to contribute. He first started by presenting the various actors involved in a contribution: a contributor, a maintainer and a reviewer. He also took the time to explain the various objectives that everyone has which is something that is often overlooked by the other parties and the conferences on this subject. He then went on to explain and document the good practices that can be used in order to contribute to most subsystems. This was overall a great overview, and we definitely recommend it to people willing to start contributing.



Real Time is Coming to Linux; What Does that Mean to You? , by Steven Rostedt

Talk selected by Michael Opdenacker

In this talk about PREEMPT_RT, the speaker, who’s a long time contributor to this feature, was approaching the subject on a new angle, taking for granted that PREEMPT_RT is in mainline Linux. That’s not quite right yet, but this is possible before the next Embedded Linux Conference, in August next year. One proof that this is on the verge of being true is that its authors no longer call it a patch set, but just PREEMPT_RT. Rostedt also added that Linux can now be called a Deterministic Operating System (aka DOS!).

So, Rostedt first explains what PREEMPT_RT is about and how it addresses the challenges of users who are determined to be deterministic (that’s my pun here, not Steven’s).

Doing this, Steven recalled the “Priority inheritance” issue that is best known through the fact that it happened on Mars on the Pathfinder robot. A high priority and critical system process got starved by a lower priority one because an even lower priority process was holding the lock the high priority process was waiting for, causing some system services to be unavailable. This caused a watchdog to kick in and reboot the system endlessly. Such an issue is addressed by “Priority inheritance”, allowing a lock-holding process to inherit the priority of the highest priority process waiting for the lock. Priority inheritance is now supported in kernel locks thanks to PREEMPT_RT.

By the way, I learned that there are now 5 preemption models in the kernel, instead of four originally with PREEMPT_RT. There is now a “Basic RT” option in which you have all the PREEMPT_RT features except the sleeping spinlocks, which is useful for debugging such features.

So now that PREEMPT_RT is almost in mainline, what should kernel developers do? The main thing is to stop adding non determinism to Linux. For example, Rostedt strongly advised against rw_locks and semaphores on multiple CPUs. That’s horrible for cache lines, as they do not scale. You should use RCU mechanisms instead.

As a kernel developer, you shouldn’t use preempt_disable() either, unless you know it is done for a very short amount of time. Similarly, if you find code that uses local_irq_save(), that’s most likely a bug. Instead, people should use spin_lock_irqsave() and spin_lock_irq(), which disable interrupts only when PREEMPT_RT is not enabled.

Rostedt ended his talk by answering a question about what will remain of the PREEMPT_RT patch set. Even when the most important parts of PREEMPT_RT are in mainline, some changesets are likely to remain for some time, just to address cases that don’t have a solution yet. 99.9% of the users will be able to do without it. That’s what a mainline solution means: no patches to apply.



Uh-oh, It’s I/O Ordering! by Will Deacon

Talk selected by Miquèl Raynal

Will gave his second talk at an ELCE about I/O ordering, 6 years after the first talk on that subject. For this purpose, he started with an introduction to the memory consistency models (in 5 minutes!) to show the audience how a very simple program, ran on two CPUs, could produce very strange results due to store buffering. Because his assumption was a bit hard to believe for such a simple program, he proved us he was right by actually running it on his laptop. While such kind of tricky behavior applies to memory, the same odd situation may happen with I/Os! After a theoretical explanation, he gave a few examples (mostly taken from the mainline Linux kernel) of good and bad code sections and explained why. If you are a device driver writer, this talk should be of interest! The examples are real use cases that you might encounter someday (if not already) and knowing how to workaround the most generic caveats with the right memory barrier or even doing a dummy read to enforce ordering is something you will want to master to avoid strange random bugs.




[PDF]

The Power Supply Subsystem, by Sebastian Reichel

Talk selected by Quentin Schulz

Sebastian started the talk by presenting what this subsystem is used for and its history, which he knows in great length since he took over the maintainership of the power supply subsystem in the Linux kernel in 2014. While it’s not the subsystem with the hardest concepts to grasp, Sebastian explained that he aimed, with his talk, at providing an accessible approach to the subsystem for people who’re trying to get started in the Linux kernel or in this specific subsystem. Having contributed to this subsystem a few patches and drivers in my early days as a kernel developer, I can say that I wish I had seen his talk before to quicken my understanding of the power supply subsystem. Scrolling down the slides, he presented a very simple example of a dummy driver, Device Tree nodes and how to configure what’s exposed to sysfs. Sebastian also gave a few words on Open-Circuit Voltage in batteries which is interesting for getting more precise values of the battery capacity depending on its age and temperature, and the ongoing work on supporting this in the kernel. He concluded with the future plans for the subsystem, which are mainly related to batteries, their fuel gauges and chargers.




[PDF]

The End of Time, 19 Years to Go, by Arnd Bergmann

Talk selected by Alexandre Belloni

Arnd gave an update on the status of the effort to get a 32-bit kernel handle the 32-bit time_t overflow which will happen in January 2038. He first started to explain why this is necessary. This boils down to the huge number of 32-bit products that are still being introduced on the market with some of them having a very long service life. Arnd said this work has been on-going since 2014, when John Stultz switched the internal timekeeping code to a 64-bit second counter. The device drivers then needed fixing. This was done by addressing them individually by changing:

  • time* to ktime_t
  • time* to jiffies
  • time_t to time64_t
  • timespec/timeval to timespec64
  • CLOCK_REALTIME to CLOCK_MONOTONIC

The driver userspace interface also needed to be changed. Some IOCTLs were easy to change because they are already using different numbers depending on the size of the argument they take. The other IOCTLs had to be redefined. It gets worse Arnd said, explaining how the read, write and mmap callbacks are getting fixed.

While the VFS layer got fixed earlier this year, some filesystems are still work in progress and other ones are not fixable because they use a 32-bit time on disk. The only way is to move away from those.

Arnd then went over the biggest remaining part of the work, the system calls. The 32-bit compat syscalls mechanism is reused and a __kernel_timespec type has been introduced to handle time at the boundary. He then listed the affected system calls and their current status.

He ended by talking about userspace and the plan to handle the issue in glibc. Finally, he mentioned what distributions will have to do.




[PDF]

On this Rock I will Build my System – Why Open-Source Firmware Matters, by Lucas Stach

Talk selected by Grégory Clement

Lucas started to present what we used to have in embedded world: a minimalist firmware which acts only as a bootloader and with no interaction with the kernel.

Then he showed why with the virtualization there were some needs to have CPU power management in a single place. This was defined by the PSCI: the purpose of it was to have the bare-metal and the virtualized kernel seeing the same interface. What should have been a simple and delimited interface then became more and more complex due to the hardware constraints. Indeed, in many SoCs multiples devices or CPUs can share the same register. Besides, an interface such as the I2C used by a PMIC can also be shared. This lead to moving the entire register inside the firmware or to have lock mechanisms between the kernel and the firmware. In conclusion, the kernel implementation became easier but at the expense of a complex firmware.

The sad news, is that most of the firmwares are not copyleft which can lead to closed source binaries, making the debugging very difficult for the kernel. Even if the firmware remains open source, having the hardware management split in two parts, makes the debugging more complex. However, there is nothing we can do about it, because there are valid reasons to have a firmware. The only thing we should be vigilant about is the openness of the firmware source.




[PDF]

Handling Security Flaws in an Open Source Project, by Jeremy Allison

Talk selected by Antoine Ténart

Samba is a well known re-implementation of the SMB protocol and as such is used in several consumer devices — such as NAS. As open source software are more and more used in new products, correctly handling security flaws and their fixes is becoming an important topic.

Jeremy Allison, one of the core developers of Samba, gave a talk about how Samba is dealing with security issues and what questions other projects should ask themselves to handle those the right way. He talked about the process to put in place to take security seriously, how to respond to vulnerability reporters and to security issues, and how to notify downstream vendors so that products in the wild are patched before the CVE is made public.

Jeremy Allison also presented three examples of security flaws in Samba. He described how they were handled at the time, the difficulties the Samba developers encountered, and gave a postmortem.

Security is important and we found this talk to be a must-see for open source maintainers and developers, as it gave a good insight on how to properly handle security vulnerabilities in a project. One of the key points was how to coordinate the security responses to avoid having the users being at risk.



[PDF]

Improve Linux User-Space Core Libraries with Restartable Sequences, by Mathieu Desnoyers

Talk selected by Maxime Chevallier

Following-up on the good LWN coverage of the restartable sequences, Mathieu Desnoyers gave an interesting talk on the current userspace support, and some feedback regarding the shortcomings of the current implementation.

Restartable sequences allow to implement lockless per-cpu sections of code, that will be automatically aborted (or restarted) whenever migration, preemption or signal delivery occurs before the final “commit” operation is done.

This is useful to read some performance counters from userspace with a minimal overhead since there’s no lock involved to protect the critical section.

Mathieu explained that these critical sections need to be written in assembly code, but thanks to the librseq and its set of macros, users shouldn’t have to worry about this.

Mathieu then presented some of the shortcomings of rseqs, one of them being that they can’t be debugged in step-by-step (since a signal interrupts the sequence, causing it to abort). To solve these shortcomings, Mathieu gave a quick glimpse of a possible new system-call, cpu_opv(), that would allow users to execute a limited sequence of instructions with preemption and migration disabled.

Power Debugging with JTAG, by Patrick Titiano & Alexandre Bailon, Baylibre

Talk selected by Thomas Petazzoni

In this talk, BayLibre engineers Patrick Titiano and Alexandre Bailon introduced libSoCCA (SoC Continuous Analyzer), a Python library that allows to watch over JTAG what a SoC is doing.

This library allows remote access to the registers of a SoC through JTAG, and uses the SoC interconnect debug port rather than the CPU debug port. Non-intrusive observation of what the SoC is doing is thus possible, even when the CPU is idle or in a low-power state.

libSoCCA uses SVD (System View Description) files, which are XML files that describe all the registers of the SoC, their bitfields and possible values. This format is not specific to libSoCCA, since it is already used by Keil, and apparently some SoC vendors provide such SVD files for their SoCs. Unfortunately, not all vendors do this, and creating such SVD files from the SoC datasheet is a very long and boring process. In addition, the speakers pointed out that the SVD file format lacked an include directive, which would be very useful to share register definitions between SoC.

With the information provided by the SVD files and a connection to the target over JTAG that uses OpenOCD, libSoCCA is then used to implement a number of different
tools:

  • PMUGraph, which shows power management statistics of the device. Compared to solution such as perf or powertop, this solution has the advantage of being non-intrusive.
  • memtool, which provides a way of manipulating registers without having to manually fiddle with register offsets and bitfields. It could be summarized as a remote devmem that knows your SoC registers. This kind of feature can be found in proprietary JTAG tools, and was lacking in the open-source world.
  • clocktool (development not started yet), which shows the state of the SoC clocks remotely, a bit like clk_summary in debugfs, but which works even when the SoC is idle or in a low power state, which is precisely a moment where getting clock status may be useful for debugging.

Overall, we found libsocca very interesting as it opens up lots of possibilities. It would be useful to have a better file format than SVD to describe SoC registers though, and it would also be nice to have an on-target variant of memtool.




[PDF]

Back from ELCE 2018: talks, tutorials and demos from Bootlin

The Embedded Linux Conference Europe edition 2018 took place last week in Edinburgh, Scotland, and no less than 9 engineers from Bootlin attended the conference. In this blog post, we would like to share the slides, materials and videos of the talks, tutorials and demos we gave during this conference.

Talk: Supporting Hardware Codecs in a Linux system – Maxime Ripard

This talk was given by Bootlin engineer Maxime Ripard, who has worked since spring 2018 with Paul Kocialkowski on adding support in the upstream Linux kernel for hardware-accelerated video decoding on Allwinner platforms. This project was the topic of the successful crowd-funding campaign we launched in February 2018, and for which we regularly posted updates on our blog.

[PDF] [Sources]


Talk: Networking: From the Ethernet MAC to the Link Partner – Maxime Chevallier & Antoine Ténart

This talk was given by Bootlin engineers Maxime Chevallier and Antoine Ténart, who shared their knowledge and experience working on enabling network hardware in Linux, trying to clarify how Ethernet MAC and PHYs interact, how PHYs communicate with their link partner, what are the protocols involved, etc.



[PDF] [Sources]

Talk: SPI Memory support in Linux and U-Boot – Miquèl Raynal

This talk was given by Miquèl Raynal, who has worked with Bootlin engineer Boris Brezillon on adding support for SPI NAND in U-Boot and Linux, as well as improving in general the support for SPI flash memory, see our previous blog post.



[PDF] [Sources]

Tutorial: Introduction to Linux kernel driver programming – Michael Opdenacker

This tutorial was given by Bootlin CEO and founder Michael Opdenacker, as part of the Embedded Apprentice Linux Engineer track.


[PDF] [Sources]

The video will be published later, as it was not recorded by the Linux Foundation, but by the E-ALE track organizers.

Tutorial: Getting started with Buildroot – Thomas Petazzoni

This tutorial was given by Bootlin CTO Thomas Petazzoni, as part of the Embedded Apprentice Linux Engineer track.



[PDF] [Sources]
The video will be published later, as it was not recorded by the Linux Foundation, but by the E-ALE track organizers.

Demo: Hardware Video Codec Support on Allwinner SoCs – Maxime Ripard

In this demonstration, Maxime Ripard was showing the upstream Linux kernel support for the Allwinner VPU, which provides hardware-accelerated video decoding for MPEG2, H264 and H265 within the Kodi media player on Allwinner platforms.



[PDF] [Sources]

Demo: Upstream Linux kernel support for Microsemi switches – Alexandre Belloni

Alexandre Belloni showing upstream Linux kernel support for Microsemi Ethernet switches
Alexandre Belloni showing upstream Linux kernel support for Microsemi Ethernet switches at the Embedded Linux Conference Europe 2018

In this demonstration, Bootlin engineer Alexandre Belloni was showing the upstream Linux kernel support for the VSC5713 and VSC7514 Microsemi Ethernet switches, which we presented in a previous blog post. Thanks to this support in upstream Linux, the different ports of the switch are seen as regular Linux network interfaces, and standard Linux user-space tools can be used to bridge the ports, set up VLAN filtering, and more. This makes such switches a lot easier to use than vendor-specific SDKs.



[Poster] [Sources]

Bootlin at the Embedded Linux Conference Europe 2018

The Embedded Linux Conference is one of the most important events in the embedded Linux industry, and Bootlin has been participating to this event non-stop since its creation in 2007. So it should be no surprise that we will once again be participating to the 2018 edition of this conference, which will take place on October 22-24 in Edinburgh, Scotland.

Embedded Linux Conference Europe 2018

In total, nine engineers from Bootlin (on a total headcount of 13!) will be attending: Alexandre Belloni, Antoine Ténart, Grégory Clement, Maxime Chevallier, Maxime Ripard, Michael Opdenacker, Miquèl Raynal, Quentin Schulz, Thomas Petazzoni.

Of course, we are not only attending, but also giving a number of talks and tutorials:

  • Monday 22 October at 11:15, Maxime Ripard is giving a talk Supporting Hardware Codecs in a Linux system, in which he will explain how HW video decoders/encoders are supported in the Video4Linux kernel subsystem, and share his experience working on the Allwinner VPU support in Linux.
  • Monday 22 October at 12:05, Antoine Ténart and Maxime Chevallier are giving a common talk Networking: From the Ethernet MAC to the Link Partner, in which they will demystify a number of acronyms and technologies used in networking hardware, and detail how Ethernet MAC and PHY are represented and managed in Linux.
  • Tuesday 23 October at 12:20, Miquèl Raynal is giving a talk SPI Memory support in Linux and U-Boot, in which he will explain how SPI NAND and SPI NOR memories are supported.
  • Thomas Petazzoni will be giving a tutorial as part of the Embedded Apprentice Linux Engineer track, titled Getting started with Buildroot. This tutorial has not been scheduled yet.
  • Michael Opdenacker will also be giving a tutorial as part of the Embedded Apprentice Linux Engineer track, titled Introduction to Linux kernel driver programming. This tutorial has not been scheduled yet.
  • We will have two demos at the Technical showcase: Alexandre Belloni will be demonstrating upstream Linux kernel support for Microsemi Ethernet switches, while Maxime Ripard will be demonstrating upstream Linux kernel support for HW accelerated video decoding on Allwinner platforms.

In addition, we will be participating to a number of co-located events:

While Bootlin CEO Michael Opdenacker was in the Embedded Linux Conference Europe Program Commitee for a number of years, he’s been replaced this year by Bootlin CTO Thomas Petazzoni. Bootlin was thus involved in the daunting but very interesting task of reviewing and selecting the talks to compose the program of this year’s event.

This is going to be a very busy week for us, and we are looking forward to attending the great talks proposed by all other speakers, and meeting the embedded Linux community once again!

Back from ELCE: participation to the Device Tree workshop

After publishing our slides and videos from the Embedded Linux Conference Europe (ELCE), reporting on talks selected by Bootlin engineers, and mentioning the award given to Michael Opdenacker, here comes the last blog post giving feedback from our participation to the 2017 edition of this conference.

On Thursday after ELCE, Bootlin engineers Maxime Ripard and Thomas Petazzoni participated to the Device Tree workshop, a day-long meeting to discuss the status and future of Device Tree support, especially in the context of the Linux kernel.

Device Tree Workshoup group photo 2017

Beyond participating to the event, Maxime and Thomas also presented briefly on two topics:

  • Maxime Ripard brought up the topic of handling foreign DT bindings (see slides). Currently, the Device Tree bindings documentation is stored in the Linux kernel source tree, in Documentation/devicetree/bindings/. However, in theory, bindings are not operating-system specific, and indeed the same bindings are used in other projects: U-Boot, Barebox, FreeBSD, Zephyr, and probably more. Maxime raised the question of what these projects should do when they create new bindings or extend existing ones? Should they contribute a patch to Linux? Should we have a separate repository for DT bindings? A bit of discussion followed, but without getting to a real conclusion.
  • Thomas Petazzoni presented on the topic of avoiding duplication in Device Tree representations (see slides). Recent Marvell Armada processors have a hardware layout where a block containing multiple IPs is duplicated several times in the SoC. In the currently available Armada 8040 there are two copies of the CP110 hardware block, and the Linux kernel carries a separate description for each. While very similar, those descriptions have subtle differences that make it non-trivial to de-duplicate. However, future SoCs will not have just 2 copies of the same hardware block, 4 copies or potentially more. In such a situation, duplicating the Device Tree description is no longer reasonable. Thomas presented a solution based on the C pre-processor, and commented on other options, such as a script to generate DTs, or improvements in the DT compiler itself. A discussion around those options followed, and while tooling improvements were considered as being the long-term solution, in the short term the solution based on the C pre-processor was acceptable upstream.

For Bootlin, participating to such events is very important, as it allows to expose to kernel developers the issue we are facing in some of our projects, and to get direct feedback from the developers on how to move forward on those topics. We definitely intend to continue participating in similar events in the future, for topics of interest to Bootlin.

Back from ELCE: award to Bootlin CEO Michael Opdenacker

The Embedded Linux Conference Europe 2017 took place at the end of October in Prague. We already posted about this event by sharing the slides and videos of Bootlin talks and later by sharing our selection of talks given by other speakers.

During the closing session of this conference, Bootlin CEO Michael Opdenacker has received from the hands of Tim Bird, on behalf of the ELCE committee, an award for its continuous participation to the Embedded Linux Conference Europe. Indeed, Michael has participated to all 11 editions of ELCE, with no interruption. He has been very active in promoting the event, especially through the video recording effort that Bootlin did in the early years of the conference, as well as through the numerous talks given by Bootlin.

Michael Opdenacker receives an award at the Embedded Linux Conference Europe

Bootlin is proud to see its continuous commitment to knowledge sharing and community participation be recognized by this award!

Back from ELCE: selection of talks from the Bootlin team

As discussed in our previous blog post, Bootlin had a strong presence at the Embedded Linux Conference Europe, with 7 attendees, 4 talks, one BoF and one poster during the technical show case.

In this blog post, we would like to highlight a number of talks from the conference that we found interesting. Each Bootlin engineer who attended the conference has selected one talk, and gives his/her feedback about this talk.

uClibc Today: Still Makes Sense – Alexey Brodkin

Talk selected by Michael Opdenacker

uClibc Today: Still Makes Sense – Alexey BrodkinAlexey Brodkin, an active contributor to the uClibc library, shared recent updates about this C library, trying to show people that the project was still active and making progress, after a few years during which it appeared to be stalled. Alexey works for Synopsys, the makers of the ARC architecture, which uClibc supports.

If you look at the repository for uClibc releases, you will see that since version 1.0.0 released in 2015, the project has made 26 releases according to a predictable schedule. The project also runs runtime regression tests on all its releases, which weren’t done before. The developers have also added support for 4 new architectures (arm64 in particular), and uClibc remains the default C library that Buildroot proposes.

Alexey highlighted that in spite of the competition from the musl library, which causes several projects to switch from uClibc to musl, uClibc still makes a lot of sense today. As a matter of fact, it supports more hardware architectures than glibc and musl do, as it’s the only one to support platforms without an MMU (such as noMMU ARM, Blackfin, m68k, Xtensa) and as the library size is still smaller that what you get with musl (though a static hello_world program is much smaller with musl if you have a close look at the library comparison tests he mentioned).

Alexey noted that the uClibc++ project is still alive too, and used in OpenWRT/Lede by default.

Read the slides and watch the video of his talk.

Identifying and Supporting “X-Compatible” hardware blocks – Chen-Yu Tsai

Talk selected by Quentin Schulz

Identifying and Supporting “X-Compatible” hardware blocks – Chen-Yu TsaiAn SoC is made of multiple IP blocks from different vendors. In some cases the source or model of the hardware blocks are neither documented nor marketed by the SoC vendor. However, since there are only very few vendors of a given IP block, stakes are high that your SoC vendor’s undocumented IP block is compatible with a known one.

With his experience in developing drivers for multiple IP blocks present in Allwinner SoCs and as a maintainer of those same SoCs, Chen-Yu first explained that SoC vendors often either embed some vendors’ licensed IP blocks in their SoCs and add the glue around it for platform- or SoC-specific hardware (clocks, resets and control signals), or they clone IP blocks with the same logic but some twists (missing, obfuscated or rearranged registers).

To identify the IP block, we can dig into the datasheet or the vendor BSP and compare those with well documented datasheets such as the one for NXP i.MX6, TI KeyStone II or the Zynq UltraScale+ MPSoC, or with mainline drivers. Asking the community is also a good idea as someone might have encountered an IP with the same behaviour before and can help us identify it quicker.

Some good identifiers for IPs could be register layouts or names along with DMA logic and descriptor format. For the unlucky ones that have been provided only a blob, they can look for the symbols in it and that may slightly help in the process.

He also mentioned that identifying an IP block is often the result of the developer’s experience in identifying IPs and other time just pure luck. Unfortunately, there are times when someone couldn’t identify the IP and wrote a complete driver just to be told by someone else in the community that this IP reminds him of that IP in which case the work done can be just thrown away. That’s where the community plays a strong role, to help us in our quest to identify an IP.

Chen-Yu then went on with the presentation of the different ways to handle the multiple variants of an IP blocks in drivers. He said that the core logic of all IP drivers is usually presented as a library and that the different variants have a driver with their own resources, extra setup and use this library. Also, a good practice is to use booleans to select features of IP blocks instead of using the ID of each variant.
For IPs whose registers have been altered, the way to go is usually to write a table for the register offsets, or use regmaps when bitfields are also modified. When the IP block differs a bit too much, custom callbacks should be used.

He ended his talk with his return from experience on multiple IP blocks (UART, USB OTG, GMAC, EMAC and HDMI) present in Allwinner SoCs and the differences developers had to handle when adding support for them.

Read the slides and watch the video of his talk.

printk(): The Most Useful Tool is Now Showing its Age – Steven Rostedt & Sergey Senozhatsky

Talks selected by Boris Brezillon. Boris also covered the related talk “printk: It’s Old, What Can We Do to Make It Young Again?” from the same speakers.

printk(): The Most Useful Tool is Now Showing its Age – Steven Rostedt & Sergey SenozhatskyMaybe I should be ashamed of saying that but printk() is one of the basic tool I’m using to debug kernel code, and I must say it did the job so far, so when I saw these presentations talking about how to improve printk() I was a bit curious. What could be wrong in printk() implementation?
Before attending the talks, I never digged into printk()’s code, because it just worked for me, but what I thought was a simple piece of code appeared to be a complex infrastructure with locking scheme that makes you realize how hard it is when several CPUs are involved.

At its core, printk() is supposed to store logs into a circular buffer and push new entries to one or several consoles. In his first talk Steven Rostedt walked through the history of printk() and explained why it became more complex when support for multi CPU appeared. He also detailed why printk() is not re-entrant and the problem it causes when called from an NMI handler. He finally went through some fixes that made the situation a bit better and advertised the 2nd half of the talk driven by Sergey Senozhatsky.

Note that between these two presentations, the printk() rework has been discussed at Kernel Summit, so Sergey already had some feedback on his proposals. While Steven presentation focused mainly on the main printk() function, Sergey gave a bit more details on why printk() can deadlock, and one of the reasons why preventing deadlocks is so complicated is that printk() delegates the ‘print to console’ aspect to console drivers which have their own locking scheme. To address that, it is proposed to move away from the callback approach and let console drivers poll for new entries in the console buffer instead, which would remove part of the locking issues. The problem with this approach is that it brings even more uncertainty on when the logs are printed on the consoles, and one of the nice things about printk() in its current form is that you are likely to have the log printed on the output before printk() returns (which helps a lot when you debug things).

He also mentioned other solutions to address other possible deadlocks, but I must admit I got lost at some point, so if you’re interested in this topic I recommend that you watch the video (printk(): The Most Useful Tool is Now Showing its Age, sadly no video is available for the second talk) and read the slides (printk(): The Most Useful Tool is Now Showing its Age and printk: It’s Old, What Can We Do to Make It Young Again?).

More robust I2C designs with a new fault-injection driver – Wolfram Sang

Talk selected by Miquèl Raynal

More robust I2C designs with a new fault-injection driver – Wolfram SangAlthough Wolfram had a lot of troubles starting its presentation lacking a proper HDMI adaptater, he gave an illuminating talk about how, as an I2C subsystem maintainer, he would like to strengthen the robustness of I2C drivers.

He first explained some basics of the I2C bus like START and STOP conditions and introduced us to a few errors he regularly spots in drivers. For instance, some badly written drivers used a START and STOP sequence while a “repeated START” was needed. This is very bad because another master on the bus could, in this little spare idle delay, decide to grab the medium and send its message. Then the message right after the repeated START would not have the expected effect. Of course plenty other errors can happen: stalled bus (SDA or SCL stuck low), lost arbitration, faulty bits… All these situations are usually due to incorrect sequences sent by the driver.

To avoid so much pain debugging obscure situations where this happens, he decided to use an extended I2C-gpio interface to access SDA and SCL from two external GPIOs and this way forces faulty situations by simply pinning high or low one line (or both) and see how the driver reacts. The I2C specification and framework provide everything to get out of a faulty situation, it is just a matter of using it (sending a STOP condition, clocking 9 times, operate a reset, etc).

Wolfram is aware of his quite conservative approach but he is really scared about breaking users by using random quirks so he tried with this talk to explain his point of view and the solutions he wants to promote.

Two questions that you might have a hard time hearing were also interesting. The first person asked if he ever considered using a “default faulty chip” designed to do by itself this kind of fault injection and see how the host reacts and behaves. Wolfram said buying hardware is too much effort for debugging, so he was more motivated to get something very easy and straightforward to use. Someone else asked if he thought about multiple clients situation, but from Wolfram’s point of view, all clients are in the same state whether the bus is busy or free and should not badly behave if we clock 9 times.

Watch the video and grab the slides.

HDMI 4k Video: Lessons Learned – Hans Verkuil

Talk selected by Maxime Ripard

HDMI 4k Video: Lessons Learned – Hans VerkuilHaving worked recently on a number of display related drivers, it was quite natural to go see what I was probably going to work on in a quite close future.

Hans started by talking about HDMI in general, and the various signals that could go through it. He then went on with what was basically a war story about all the mistakes, gotchas and misconceptions that he encountered while working on a video-conference box for Cisco. He covered the hardware itself, but also more low-level oriented aspects, such as the clocks frequencies needed to operate properly, the various signals you could look at for debugging or the issues that might come with the associated encoding and / or color spaces, especially when you want to support as many displays as you can. He also pointed out the flaws in the specifications that might lead to implementation inconsistencies. He concluded with the flaws of various HDMI adapters, the issues that might arise using them on various OSes, and how to work around them when doable.

Watch the video and the slides.

The Serial Device Bus – Johan Hovold

Talk selected by Thomas Petazzoni

The Serial Device Bus – Johan HovoldJohan started his talk at ELCE by exposing the problem with how serial ports (UARTs) are currently handled in the Linux kernel. Serial ports are handled by the TTY layer, which allows user-space applications to send and receive data with what is connected on the other side of the UART. However, the kernel doesn’t provide a good mechanism to model the device that is connected at the other side of the UART, such as a Bluetooth chip. Due to this, people have resorted to either writing user-space drivers for such devices (which falls short when those devices need additional resources such as regulators, GPIOs, etc.) or to developing specific TTY line-discipline in the kernel. The latter also doesn’t work very well because a line discipline needs to be explicitly attached to a UART to operate, which requires a user-space program such as hciattach used in Bluetooth applications.

In order to address this problem, Johan picked up the work initially started by Rob Herring (Linaro), which consisted in writing a serial device bus (serdev in short), which consists in turning UART into a proper bus, with bus controllers (UART controllers) and devices connected to this bus, very much like other busses in the Linux kernel (I2C, SPI, etc.). serdev was initially merged in Linux 4.11, with some improvements being merged in follow-up versions. Johan then described in details how serdev works. First, there is a TTY port controller, which instead of registering the traditional TTY character device will register the serdev controller and its slave devices. Thanks to this, one can described in its Device Tree child nodes of the UART controller node, which will be probed as serdev slaves. There is then a serdev API to allow the implementation of serdev slave drivers, that can send and receive data of the UART. Already a few drivers are using this API: hci_serdev, hci_bcm, hci_ll, hci_nokia (Bluetooth) and qca_uart (Ethernet).

We found this talk very interesting, as it clearly explained what is the use case for serdev and how it works, and it should become a very useful subsystem for many embedded applications that use UART-connected devices.

Watch the video and the slides.

GStreamer for Tiny Devices – Olivier Crête

Talk selected by Grégory Clement

GStreamer for Tiny Devices – Olivier CrêteThe purpose of this talk was to show how to shrink Gstreamer to make it fit in an embedded Linux device. First, Olivier Crête introduced what GStreamer is, it was very high level but well done. Then after presenting the issue, he showed step by step how he managed to reduce the footprint of a GStreamer application to fit in his device.

In a first part it was a focus on features specific to GStreamer such as how to generate only the needed plugins. Then most of the tricks showed could be used for any C or C++ application. The talk was pretty short so there was no useless or boring part Moreover, the speaker himself was good and dynamic.

To conclude it was a very pleasant talk teaching step by step how to reduce the footprint of an application being GStreamer or not.

Watch the video and the slides.