Author: Thomas Petazzoni

The Embedded Linux Conference Europe edition 2018 took place a few weeks ago in Edinburgh, Scotland, and no less than 9 engineers from Bootlin attended the conference. While our previous blog post shared the videos and slides of our talks, tutorials and demos, in this blog post we would like to highlight a selection of talks that Bootlin engineers found interesting. We asked each of the 9 engineers who attended the event to pick one talk they liked, and make a small write-up about it. Of course, many other talks were interesting and what makes a talk interesting is very subjective!

Getting Your Patches in Mainline Linux: What Not To Do (and a Few Things You Could Try Instead), by Marc Zyngier

Talk selected by Maxime Ripard

Marc gave a talk on a subject that is often debated, and still confusing to newcomers: how to contribute. He first started by presenting the various actors involved in a contribution: a contributor, a maintainer and a reviewer. He also took the time to explain the various objectives that everyone has which is something that is often overlooked by the other parties and the conferences on this subject. He then went on to explain and document the good practices that can be used in order to contribute to most subsystems. This was overall a great overview, and we definitely recommend it to people willing to start contributing.

Real Time is Coming to Linux; What Does that Mean to You? , by Steven Rostedt

Talk selected by Michael Opdenacker

In this talk about PREEMPT_RT, the speaker, who’s a long time contributor to this feature, was approaching the subject on a new angle, taking for granted that PREEMPT_RT is in mainline Linux. That’s not quite right yet, but this is possible before the next Embedded Linux Conference, in August next year. One proof that this is on the verge of being true is that its authors no longer call it a patch set, but just PREEMPT_RT. Rostedt also added that Linux can now be called a Deterministic Operating System (aka DOS!).

So, Rostedt first explains what PREEMPT_RT is about and how it addresses the challenges of users who are determined to be deterministic (that’s my pun here, not Steven’s).

Doing this, Steven recalled the “Priority inheritance” issue that is best known through the fact that it happened on Mars on the Pathfinder robot. A high priority and critical system process got starved by a lower priority one because an even lower priority process was holding the lock the high priority process was waiting for, causing some system services to be unavailable. This caused a watchdog to kick in and reboot the system endlessly. Such an issue is addressed by “Priority inheritance”, allowing a lock-holding process to inherit the priority of the highest priority process waiting for the lock. Priority inheritance is now supported in kernel locks thanks to PREEMPT_RT.

By the way, I learned that there are now 5 preemption models in the kernel, instead of four originally with PREEMPT_RT. There is now a “Basic RT” option in which you have all the PREEMPT_RT features except the sleeping spinlocks, which is useful for debugging such features.

So now that PREEMPT_RT is almost in mainline, what should kernel developers do? The main thing is to stop adding non determinism to Linux. For example, Rostedt strongly advised against rw_locks and semaphores on multiple CPUs. That’s horrible for cache lines, as they do not scale. You should use RCU mechanisms instead.

As a kernel developer, you shouldn’t use preempt_disable() either, unless you know it is done for a very short amount of time. Similarly, if you find code that uses local_irq_save(), that’s most likely a bug. Instead, people should use spin_lock_irqsave() and spin_lock_irq(), which disable interrupts only when PREEMPT_RT is not enabled.

Rostedt ended his talk by answering a question about what will remain of the PREEMPT_RT patch set. Even when the most important parts of PREEMPT_RT are in mainline, some changesets are likely to remain for some time, just to address cases that don’t have a solution yet. 99.9% of the users will be able to do without it. That’s what a mainline solution means: no patches to apply.

Uh-oh, It’s I/O Ordering! by Will Deacon

Talk selected by Miquèl Raynal

Will gave his second talk at an ELCE about I/O ordering, 6 years after the first talk on that subject. For this purpose, he started with an introduction to the memory consistency models (in 5 minutes!) to show the audience how a very simple program, ran on two CPUs, could produce very strange results due to store buffering. Because his assumption was a bit hard to believe for such a simple program, he proved us he was right by actually running it on his laptop. While such kind of tricky behavior applies to memory, the same odd situation may happen with I/Os! After a theoretical explanation, he gave a few examples (mostly taken from the mainline Linux kernel) of good and bad code sections and explained why. If you are a device driver writer, this talk should be of interest! The examples are real use cases that you might encounter someday (if not already) and knowing how to workaround the most generic caveats with the right memory barrier or even doing a dummy read to enforce ordering is something you will want to master to avoid strange random bugs.

The Power Supply Subsystem, by Sebastian Reichel

Talk selected by Quentin Schulz

Sebastian started the talk by presenting what this subsystem is used for and its history, which he knows in great length since he took over the maintainership of the power supply subsystem in the Linux kernel in 2014. While it’s not the subsystem with the hardest concepts to grasp, Sebastian explained that he aimed, with his talk, at providing an accessible approach to the subsystem for people who’re trying to get started in the Linux kernel or in this specific subsystem. Having contributed to this subsystem a few patches and drivers in my early days as a kernel developer, I can say that I wish I had seen his talk before to quicken my understanding of the power supply subsystem. Scrolling down the slides, he presented a very simple example of a dummy driver, Device Tree nodes and how to configure what’s exposed to sysfs. Sebastian also gave a few words on Open-Circuit Voltage in batteries which is interesting for getting more precise values of the battery capacity depending on its age and temperature, and the ongoing work on supporting this in the kernel. He concluded with the future plans for the subsystem, which are mainly related to batteries, their fuel gauges and chargers.

The End of Time, 19 Years to Go, by Arnd Bergmann

Talk selected by Alexandre Belloni

Arnd gave an update on the status of the effort to get a 32-bit kernel handle the 32-bit time_t overflow which will happen in January 2038. He first started to explain why this is necessary. This boils down to the huge number of 32-bit products that are still being introduced on the market with some of them having a very long service life. Arnd said this work has been on-going since 2014, when John Stultz switched the internal timekeeping code to a 64-bit second counter. The device drivers then needed fixing. This was done by addressing them individually by changing:

• time* to ktime_t
• time* to jiffies
• time_t to time64_t
• timespec/timeval to timespec64
• CLOCK_REALTIME to CLOCK_MONOTONIC

The driver userspace interface also needed to be changed. Some IOCTLs were easy to change because they are already using different numbers depending on the size of the argument they take. The other IOCTLs had to be redefined. It gets worse Arnd said, explaining how the read, write and mmap callbacks are getting fixed.

While the VFS layer got fixed earlier this year, some filesystems are still work in progress and other ones are not fixable because they use a 32-bit time on disk. The only way is to move away from those.

Arnd then went over the biggest remaining part of the work, the system calls. The 32-bit compat syscalls mechanism is reused and a __kernel_timespec type has been introduced to handle time at the boundary. He then listed the affected system calls and their current status.

He ended by talking about userspace and the plan to handle the issue in glibc. Finally, he mentioned what distributions will have to do.

On this Rock I will Build my System – Why Open-Source Firmware Matters, by Lucas Stach

Talk selected by Grégory Clement

Lucas started to present what we used to have in embedded world: a minimalist firmware which acts only as a bootloader and with no interaction with the kernel.

Then he showed why with the virtualization there were some needs to have CPU power management in a single place. This was defined by the PSCI: the purpose of it was to have the bare-metal and the virtualized kernel seeing the same interface. What should have been a simple and delimited interface then became more and more complex due to the hardware constraints. Indeed, in many SoCs multiples devices or CPUs can share the same register. Besides, an interface such as the I2C used by a PMIC can also be shared. This lead to moving the entire register inside the firmware or to have lock mechanisms between the kernel and the firmware. In conclusion, the kernel implementation became easier but at the expense of a complex firmware.

The sad news, is that most of the firmwares are not copyleft which can lead to closed source binaries, making the debugging very difficult for the kernel. Even if the firmware remains open source, having the hardware management split in two parts, makes the debugging more complex. However, there is nothing we can do about it, because there are valid reasons to have a firmware. The only thing we should be vigilant about is the openness of the firmware source.

Handling Security Flaws in an Open Source Project, by Jeremy Allison

Talk selected by Antoine Ténart

Samba is a well known re-implementation of the SMB protocol and as such is used in several consumer devices — such as NAS. As open source software are more and more used in new products, correctly handling security flaws and their fixes is becoming an important topic.

Jeremy Allison, one of the core developers of Samba, gave a talk about how Samba is dealing with security issues and what questions other projects should ask themselves to handle those the right way. He talked about the process to put in place to take security seriously, how to respond to vulnerability reporters and to security issues, and how to notify downstream vendors so that products in the wild are patched before the CVE is made public.

Jeremy Allison also presented three examples of security flaws in Samba. He described how they were handled at the time, the difficulties the Samba developers encountered, and gave a postmortem.

Security is important and we found this talk to be a must-see for open source maintainers and developers, as it gave a good insight on how to properly handle security vulnerabilities in a project. One of the key points was how to coordinate the security responses to avoid having the users being at risk.

Improve Linux User-Space Core Libraries with Restartable Sequences, by Mathieu Desnoyers

Talk selected by Maxime Chevallier

Following-up on the good LWN coverage of the restartable sequences, Mathieu Desnoyers gave an interesting talk on the current userspace support, and some feedback regarding the shortcomings of the current implementation.

Restartable sequences allow to implement lockless per-cpu sections of code, that will be automatically aborted (or restarted) whenever migration, preemption or signal delivery occurs before the final “commit” operation is done.

This is useful to read some performance counters from userspace with a minimal overhead since there’s no lock involved to protect the critical section.

Mathieu explained that these critical sections need to be written in assembly code, but thanks to the librseq and its set of macros, users shouldn’t have to worry about this.

Mathieu then presented some of the shortcomings of rseqs, one of them being that they can’t be debugged in step-by-step (since a signal interrupts the sequence, causing it to abort). To solve these shortcomings, Mathieu gave a quick glimpse of a possible new system-call, cpu_opv(), that would allow users to execute a limited sequence of instructions with preemption and migration disabled.

Power Debugging with JTAG, by Patrick Titiano & Alexandre Bailon, Baylibre

Talk selected by Thomas Petazzoni

In this talk, BayLibre engineers Patrick Titiano and Alexandre Bailon introduced libSoCCA (SoC Continuous Analyzer), a Python library that allows to watch over JTAG what a SoC is doing.

This library allows remote access to the registers of a SoC through JTAG, and uses the SoC interconnect debug port rather than the CPU debug port. Non-intrusive observation of what the SoC is doing is thus possible, even when the CPU is idle or in a low-power state.

libSoCCA uses SVD (System View Description) files, which are XML files that describe all the registers of the SoC, their bitfields and possible values. This format is not specific to libSoCCA, since it is already used by Keil, and apparently some SoC vendors provide such SVD files for their SoCs. Unfortunately, not all vendors do this, and creating such SVD files from the SoC datasheet is a very long and boring process. In addition, the speakers pointed out that the SVD file format lacked an include directive, which would be very useful to share register definitions between SoC.

With the information provided by the SVD files and a connection to the target over JTAG that uses OpenOCD, libSoCCA is then used to implement a number of different
tools:

• PMUGraph, which shows power management statistics of the device. Compared to solution such as perf or powertop, this solution has the advantage of being non-intrusive.
• memtool, which provides a way of manipulating registers without having to manually fiddle with register offsets and bitfields. It could be summarized as a remote devmem that knows your SoC registers. This kind of feature can be found in proprietary JTAG tools, and was lacking in the open-source world.
• clocktool (development not started yet), which shows the state of the SoC clocks remotely, a bit like clk_summary in debugfs, but which works even when the SoC is idle or in a low power state, which is precisely a moment where getting clock status may be useful for debugging.

Overall, we found libsocca very interesting as it opens up lots of possibilities. It would be useful to have a better file format than SVD to describe SoC registers though, and it would also be nice to have an on-target variant of memtool.

The Embedded Linux Conference Europe edition 2018 took place last week in Edinburgh, Scotland, and no less than 9 engineers from Bootlin attended the conference. In this blog post, we would like to share the slides, materials and videos of the talks, tutorials and demos we gave during this conference.

Talk: Supporting Hardware Codecs in a Linux system – Maxime Ripard

This talk was given by Bootlin engineer Maxime Ripard, who has worked since spring 2018 with Paul Kocialkowski on adding support in the upstream Linux kernel for hardware-accelerated video decoding on Allwinner platforms. This project was the topic of the successful crowd-funding campaign we launched in February 2018, and for which we regularly posted updates on our blog.

Talk: Networking: From the Ethernet MAC to the Link Partner – Maxime Chevallier & Antoine Ténart

This talk was given by Bootlin engineers Maxime Chevallier and Antoine Ténart, who shared their knowledge and experience working on enabling network hardware in Linux, trying to clarify how Ethernet MAC and PHYs interact, how PHYs communicate with their link partner, what are the protocols involved, etc.

Talk: SPI Memory support in Linux and U-Boot – Miquèl Raynal

This talk was given by Miquèl Raynal, who has worked with Bootlin engineer Boris Brezillon on adding support for SPI NAND in U-Boot and Linux, as well as improving in general the support for SPI flash memory, see our previous blog post.

Tutorial: Introduction to Linux kernel driver programming – Michael Opdenacker

This tutorial was given by Bootlin CEO and founder Michael Opdenacker, as part of the Embedded Apprentice Linux Engineer track.

The video will be published later, as it was not recorded by the Linux Foundation, but by the E-ALE track organizers.

Tutorial: Getting started with Buildroot – Thomas Petazzoni

This tutorial was given by Bootlin CTO Thomas Petazzoni, as part of the Embedded Apprentice Linux Engineer track.

Demo: Hardware Video Codec Support on Allwinner SoCs – Maxime Ripard

In this demonstration, Maxime Ripard was showing the upstream Linux kernel support for the Allwinner VPU, which provides hardware-accelerated video decoding for MPEG2, H264 and H265 within the Kodi media player on Allwinner platforms.

[PDF] [Sources]

Demo: Upstream Linux kernel support for Microsemi switches – Alexandre Belloni

In this demonstration, Bootlin engineer Alexandre Belloni was showing the upstream Linux kernel support for the VSC5713 and VSC7514 Microsemi Ethernet switches, which we presented in a previous blog post. Thanks to this support in upstream Linux, the different ports of the switch are seen as regular Linux network interfaces, and standard Linux user-space tools can be used to bridge the ports, set up VLAN filtering, and more. This makes such switches a lot easier to use than vendor-specific SDKs.

[Poster] [Sources]

The Embedded Linux Conference is one of the most important events in the embedded Linux industry, and Bootlin has been participating to this event non-stop since its creation in 2007. So it should be no surprise that we will once again be participating to the 2018 edition of this conference, which will take place on October 22-24 in Edinburgh, Scotland.

In total, nine engineers from Bootlin (on a total headcount of 13!) will be attending: Alexandre Belloni, Antoine Ténart, Grégory Clement, Maxime Chevallier, Maxime Ripard, Michael Opdenacker, Miquèl Raynal, Quentin Schulz, Thomas Petazzoni.

Of course, we are not only attending, but also giving a number of talks and tutorials:

• Monday 22 October at 11:15, Maxime Ripard is giving a talk Supporting Hardware Codecs in a Linux system, in which he will explain how HW video decoders/encoders are supported in the Video4Linux kernel subsystem, and share his experience working on the Allwinner VPU support in Linux.
• Monday 22 October at 12:05, Antoine Ténart and Maxime Chevallier are giving a common talk Networking: From the Ethernet MAC to the Link Partner, in which they will demystify a number of acronyms and technologies used in networking hardware, and detail how Ethernet MAC and PHY are represented and managed in Linux.
• Tuesday 23 October at 12:20, Miquèl Raynal is giving a talk SPI Memory support in Linux and U-Boot, in which he will explain how SPI NAND and SPI NOR memories are supported.
• Thomas Petazzoni will be giving a tutorial as part of the Embedded Apprentice Linux Engineer track, titled Getting started with Buildroot. This tutorial has not been scheduled yet.
• Michael Opdenacker will also be giving a tutorial as part of the Embedded Apprentice Linux Engineer track, titled Introduction to Linux kernel driver programming. This tutorial has not been scheduled yet.
• We will have two demos at the Technical showcase: Alexandre Belloni will be demonstrating upstream Linux kernel support for Microsemi Ethernet switches, while Maxime Ripard will be demonstrating upstream Linux kernel support for HW accelerated video decoding on Allwinner platforms.

In addition, we will be participating to a number of co-located events:

• Thomas Petazzoni will participate to the Buildroot Developers Meeting, on October 20-21
• Maxime Ripard will participate to the Linux Media Summit, on October 25
• Maxime Chevallier will participate to the Real Time Summit, also on October 25

While Bootlin CEO Michael Opdenacker was in the Embedded Linux Conference Europe Program Commitee for a number of years, he’s been replaced this year by Bootlin CTO Thomas Petazzoni. Bootlin was thus involved in the daunting but very interesting task of reviewing and selecting the talks to compose the program of this year’s event.

This is going to be a very busy week for us, and we are looking forward to attending the great talks proposed by all other speakers, and meeting the embedded Linux community once again!