Category: Technical

As we reported in a previous blog post, almost the entire Bootlin engineering team was at the Embedded Linux Conference Europe in Prague in June. In order to share with our readers more about what happened at this conference, we have asked all engineers at Bootlin to select one talk they found interesting and useful and share a short summary of it. We will share this feedback in a series of blog post: first post, second post, this one being the third of the series.

rtla timerlat: Debugging Real-time Linux Scheduling Latency

Talk by Daniel Bristot de Oliveira, chosen by Bootlin engineer Maxime Chevallier.

Talks related to real-time linux debugging are pretty common at ELCE, I gave one myself in 2017 and I’ve been attending most of them since then. Besides a headache, what I could get from attending all these talks is that this topic is complex, time consuming, and that there’s a lot of different methodologies one can use to find the cause of these elusive problems.

Users who aren’t very familiar with the inner workings of the Linux Kernel can ask for help on mailing-lists, and the reply usually asks for a trace. This is where things get complicated, the Linux kernel tracer is very powerful, but can drown users in a flood of trace events from which it is difficult to extract the relevant data.

Hopefully, Daniel’s talk is going to make this kind of talk less common, as the tool he wrote and presented, rtla, makes it easy to gather important information about the cause of undesired latencies. By using cleverly placed trace-points, in-kernel testing tools (timerlat and osnoise) and an automated trace analyzer, rtla can not only detect latencies as cyclictest would, it can also give you what caused the latency. If it’s a blocking problem, rtla tells you which process is blocking your task. If it’s an interference, rtla will tell you which task or interrupt caused the latency, and can even detect if the hardware itself is the culprit.

For developers, this tool is also a perfect way to gather user feedback and bug reports that are small, precise and easily reproducible.

I therefore strongly recommend checking out Daniel’s talk and his dedicated blog article.

Slides: PDF
Video: Youtube

Zbus – the Lightweight and Flexible Zephyr Message Bus

Talk by Robrigo Peixoto, chosen by Bootlin engineer Thomas Perrot

Zbus is a new message bus for Zephyr allowing threads to communicate to many others, easily. This bus allows to implement several bus topologies:

• • one-to-one
  • one-to-many
  • Many-to-many

In addition, it can be used on very constrained systems.

In this talk, Rodrigo explained in detail how Zbus works, through a few examples. A thread can read or publish in bus channels, and when a message is published into a channel:

• • • The Listener’s callbacks are executed
    • A notification is put to the subscriber’s queues
    • Then the subscriber will be executed by priority order

The bus is managed by a dispatcher, named Virtual Distributed Event Dispatcher (VDED) that is robust to priority inversion.

We found Zbus to be a very interesting feature because before there was no easy way to implement one-to-many and many-to-many topologies, but also one-to-one communications without having to manage the problems of inverting priorities and to use FIFO, LIFO, pipe, etc.

Slides: PDF
Video: Youtube

Linux Power ! (from the Perspective of a PMIC Vendor)

Talk by Matti Vaittinen, chosen by Bootlin engineer Kamel Bouhara.

PMICs (Power Management Integrated Circuit) are a key component of low power embedded systems as they often handle complexity in controlling various power voltages required by SoCs. In his talk Matti Vaittinen started by depicting the various devices that can be embedded in a PMIC (Power Management Integrated Circuit): watchdog, RTC, GPIOs are examples of such extra functionalities. He reminded us the reason why such devices are best fitted in the Linux MFD subsystem to take advantage of existing code. However the main subsystem used to implement support for a PMIC is the regulator subsystem and the talk gives us a good understanding of how it works, the concept of provider/consumer, how to register multiple regulators for a PMIC and how to handle specific events. A focus is made on error detection and how over current errors are reported over three categories:

• • • PROTECTION : hardware level errors reported when protection limit is reached
    • ERROR: Unrecoverable errors that don’t directly involve hardware shutdown.
    • WARNING: System is still recoverable but requires specific action to be taken

Some PMICs also provide IRQs to notify errors or events and the kernel provides a helper function to handle such notifications and map them to specific actions depending on their severity.

Overall, we found this talk interesting to understand bettert the features provided by PMICs, and how these features are supported by Linux.

Slides: PDF
Video: Youtube