Real-Time Summit 2018

This year, Bootlin engineer Maxime Chevallier attended the Real-Time Summit, which took place after ELCE in Edinburgh. In a similar fashion to the Linux Media Summit, this was a 1-day conference dedicated to Real-Time topics in the Linux ecosystem, more specifically the Linux Kernel.

The future of Preempt-RT

Real-Time (or should we say, deterministic) behavior in the Linux kernel has been pursued for a long time, the most famous effort being the Preempt-RT patch. As Steven Rostedt announced during his talk at ELCE 2018, the Preempt-RT patch is close to being fully merged in mainline Linux, we can expect to see this happen in 2019.

Some of the maintainers of the Preempt-RT patch were present at the Real-Time summit, including Thomas Gleixner who lead the discussion throughout the day.

This was the occasion to discuss the remaining points to be addressed for Preempt-RT to make it into mainline Linux :

  • Printk : As Steven Rostedt explained at ELCE 2017, printk is not very real-time friendly. The main issue was worked around, but John Ogness presented his current work of fully redesigning printk’s behaviour.
  • Thomas Gleixner talked about the current state of softirq handling, which is also a critical point for determinism. They work by “stealing” some irq context time, falling back to ksoftirqd when necessary. This is particularly problematic for networking drivers that heavily rely on softirq.
  • Peter Zijlstra exposed the different scheduler related issues that needs to be addressed, focusing on SCHED_DEADLINE.

Modeling and analyzing the kernel behavior

All the talks weren’t about the Preempt-RT match merging effort. Daniel Bristot de Oliveira presented his ongoing academic work on modeling the Linux task model. The idea here is to build a formal model that doesn’t take shortcuts or idealize the way tasks are handled in the kernel, so that this can be used as a basis for academic research on topics such as scheduling.

One of the main arguments is that there’s a gap in terms of language and methodology used between kernel developers and the academic world. Daniel explained how he managed to build a huge state-machine representing the task model, and how he uses it now to verify that tasks behave how they should by running trace events in the state machine.

This talk sparked a lot if interesting discussions, for example Peter Zijlstra suggested to compile the state machine into eBPF code and run it live in the kernel.

Julia Lawall was present in the room, and improvised a talk inspired by Daniel’s presentation. She presented DSAC, a static analysis tool dedicated to finding Sleeping in Atomic Context bugs. Julia is involved in the development and use of the coccinelle tool, and explained that it is quickly limited when trying to find that categories of bugs, where sleeping calls can be deeply nested in a call stack protected by spinlocks. Using LLVM, DSAC can analyze complex scenarios with multiple level of nesting and indirect calls to detect SAC bugs. After analyzing the v4.17 kernel sources for only a few hours, the tool was able to detect more than 1000 bugs, 220 of which were confirmed.

The overall technical level of the different talks was high, leading to passionate discussions and suggestions on every topic that was brought during the day.

Xenomai 2.6.4 released, with Atmel SAMA5D3 support

XenomaiAt the end of September, the Xenomai project has announced the release of version 2.6.4. For the record, Xenomai is a hard real-time extension to the Linux kernel.

Amongst a number of bug fixes and improvements, this new release brings an interesting new feature to which Bootlin contributed: the support for the Atmel SAMA5D3. This means that Xenomai can now be used on platforms such as the Xplained SAMA5D3 and any other SAMA5D3 based platform. This work was done by Xenomai ARM maintainer Gilles Chanteperdrix, thanks to the testing and insights of Bootlin engineer Maxime Ripard.

Mainly, the change needed was to support the AIC5 interrupt controller used in SAMA5D3, which is different from the interrupt controller used on earlier AT91 processors. This change should also provide compatibility with the recently released SAMA5D4, though we haven’t tested this at this time, and Xenomai only provides its patch up to kernel 3.14, while SAMA5D4 support was only recently added to the mainline kernel.

This 2.6.4 Xenomai release also brings support for the 3.14 kernel version, through the corresponding I-Pipe patch.

There are also some other interesting Xenomai news: in early October, they have released the first release candidate of Xenomai 3, the next generation Xenomai architecture. And they also have a brand new and more modern website at xenomai.org.

Report on extensive real-time Linux benchmarks on AT91

The real time page I wrote for Atmel was finally released on the Linux4Sam Atmel Wiki. The purpose of this page was to help new comers to use real time features with Atmel CPUs and to present the state of the real time support.

Here are some figures associated to this work:

  • On this page I present the results of more than 300 hours of benchmarks!
  • During the setup and the tuning tests ran for more than 600 hours.
  • Analysis and formatting took a few dozen hours of work.
  • The benchmarks have been run on 3 boards, 3 flavors of Linux (vanilla, PREEMPT-RT patches, Xenomai co-kernel approach), and 2 kinds of tests (timer-based and GPIO-based)