New training course: Real-Time Linux with PREEMPT_RT

In the field of embedded systems, a number of applications need real-time guarantees, and the Linux ecosystem has been offering for a long time a number of solutions to address those needs, either by improving the Linux kernel itself using the PREEMPT_RT approach, or by using a co-kernel approach such as the one offered by Xenomai. Bootlin training’s portfolio already has an initial coverage of these topics in our Embedded Linux system development course.

Today, we are happy to announce a brand new Real-Time Linux with PREEMPT_RT, which is specifically focused on the PREEMPT_RT solution. This solution made a vast amount of progress in recent times in terms of integration into the official Linux kernel, which makes it even more relevant for a number of projects which need real-time guarantees.

The main topics covered by the course are:

What is a real-time and deterministic operating system
How to configure, build and setup a PREEMPT_RT enabled Linux kernel
How to identify and benchmark the hardware platform in terms of real-time characteristics
How to configure and tune the Linux kernel and the system for deterministic behavior
How to develop and debug real-time user-space Linux applications as well as analyze latencies

The course is illustrated by practical labs or demonstrations made on the BeagleBone Black platform. It has been developed and is taught by Bootlin engineer Maxime Chevallier, who is an experienced embedded Linux and Linux kernel engineer and trainer.

As usual our training materials are fully open-source, including the ones for this brand new session. You can read the Slides and Practical lab instructions. Bootlin is one of the very few companies delivering training courses to make its training materials open-source: by choosing to work with Bootlin for your trainings, you support our work on developing and publishing freely available training materials.

If you’re interested in getting this new Real-Time Linux with PREEMPT_RT training course, you have three options:

Public on-line sessions, opened to individual registration. The course lasts 3 sessions of 4 hours. We have scheduled a first session on January 19-21, 2022, and registration is open, for 399 EUR at the Early bird rate, of 449 EUR at the Regular rate.
Dedicated on-line sessions, which we organize at the date/time of your choice. The course also lasts 3 sessions of 4 hours. Contact us to request a quote.
Dedicated on-site sessions, where our trainer travels to your location to deliver the training course. In this case, the course lasts two full days. Contact us to request a quote.

Real-Time Summit 2018

This year, Bootlin engineer Maxime Chevallier attended the Real-Time Summit, which took place after ELCE in Edinburgh. In a similar fashion to the Linux Media Summit, this was a 1-day conference dedicated to Real-Time topics in the Linux ecosystem, more specifically the Linux Kernel.

The future of Preempt-RT

Real-Time (or should we say, deterministic) behavior in the Linux kernel has been pursued for a long time, the most famous effort being the Preempt-RT patch. As Steven Rostedt announced during his talk at ELCE 2018, the Preempt-RT patch is close to being fully merged in mainline Linux, we can expect to see this happen in 2019.

Some of the maintainers of the Preempt-RT patch were present at the Real-Time summit, including Thomas Gleixner who lead the discussion throughout the day.

This was the occasion to discuss the remaining points to be addressed for Preempt-RT to make it into mainline Linux :

Printk : As Steven Rostedt explained at ELCE 2017, printk is not very real-time friendly. The main issue was worked around, but John Ogness presented his current work of fully redesigning printk’s behaviour.
Thomas Gleixner talked about the current state of softirq handling, which is also a critical point for determinism. They work by “stealing” some irq context time, falling back to ksoftirqd when necessary. This is particularly problematic for networking drivers that heavily rely on softirq.
Peter Zijlstra exposed the different scheduler related issues that needs to be addressed, focusing on SCHED_DEADLINE.

Modeling and analyzing the kernel behavior

All the talks weren’t about the Preempt-RT match merging effort. Daniel Bristot de Oliveira presented his ongoing academic work on modeling the Linux task model. The idea here is to build a formal model that doesn’t take shortcuts or idealize the way tasks are handled in the kernel, so that this can be used as a basis for academic research on topics such as scheduling.

One of the main arguments is that there’s a gap in terms of language and methodology used between kernel developers and the academic world. Daniel explained how he managed to build a huge state-machine representing the task model, and how he uses it now to verify that tasks behave how they should by running trace events in the state machine.

This talk sparked a lot if interesting discussions, for example Peter Zijlstra suggested to compile the state machine into eBPF code and run it live in the kernel.

Julia Lawall was present in the room, and improvised a talk inspired by Daniel’s presentation. She presented DSAC, a static analysis tool dedicated to finding Sleeping in Atomic Context bugs. Julia is involved in the development and use of the coccinelle tool, and explained that it is quickly limited when trying to find that categories of bugs, where sleeping calls can be deeply nested in a call stack protected by spinlocks. Using LLVM, DSAC can analyze complex scenarios with multiple level of nesting and indirect calls to detect SAC bugs. After analyzing the v4.17 kernel sources for only a few hours, the tool was able to detect more than 1000 bugs, 220 of which were confirmed.

The overall technical level of the different talks was high, leading to passionate discussions and suggestions on every topic that was brought during the day.

Xenomai 2.6.4 released, with Atmel SAMA5D3 support

At the end of September, the Xenomai project has announced the release of version 2.6.4. For the record, Xenomai is a hard real-time extension to the Linux kernel.

Amongst a number of bug fixes and improvements, this new release brings an interesting new feature to which Bootlin contributed: the support for the Atmel SAMA5D3. This means that Xenomai can now be used on platforms such as the Xplained SAMA5D3 and any other SAMA5D3 based platform. This work was done by Xenomai ARM maintainer Gilles Chanteperdrix, thanks to the testing and insights of Bootlin engineer Maxime Ripard.

Mainly, the change needed was to support the AIC5 interrupt controller used in SAMA5D3, which is different from the interrupt controller used on earlier AT91 processors. This change should also provide compatibility with the recently released SAMA5D4, though we haven’t tested this at this time, and Xenomai only provides its patch up to kernel 3.14, while SAMA5D4 support was only recently added to the mainline kernel.

This 2.6.4 Xenomai release also brings support for the 3.14 kernel version, through the corresponding I-Pipe patch.

There are also some other interesting Xenomai news: in early October, they have released the first release candidate of Xenomai 3, the next generation Xenomai architecture. And they also have a brand new and more modern website at xenomai.org.

Report on extensive real-time Linux benchmarks on AT91

The real time page I wrote for Atmel was finally released on the Linux4Sam Atmel Wiki. The purpose of this page was to help new comers to use real time features with Atmel CPUs and to present the state of the real time support.

Here are some figures associated to this work:

On this page I present the results of more than 300 hours of benchmarks!

During the setup and the tuning tests ran for more than 600 hours.

Analysis and formatting took a few dozen hours of work.

The benchmarks have been run on 3 boards, 3 flavors of Linux (vanilla, PREEMPT-RT patches, Xenomai co-kernel approach), and 2 kinds of tests (timer-based and GPIO-based)