performance – Bootlin

Since its inception, Bootlin has offered training courses on technical topics related to the use of Linux in embedded systems, with freely accessible training materials, and trainers with in-depth and real-life experience in their field. Based on these ideas, we have progressively extended our training portfolio over the years.

Today, we are extremely happy to announce a brand new course: Linux debugging, profiling, tracing and performance analysis. Many of our customers working on embedded Linux systems have expressed interest in diving into these complex topics, so we’ve created this course to:

Give enough background information about how Linux works to be able to have a solid reasoning when investigating performance issues or simply bugs. Our course therefore details how user-space vs. kernel-space works, scheduling, and memory management, as a prerequisite to understanding better how Linux works.
Give a strong introduction to the most important debugging, profiling and tracing tools in Linux, which are often not easy to get started with. We cover a very large spectrum of tools: strace, gdb, perf, ftrace, LTTng, kgdb, kmemleak, and many more. All these tools are illustrated through practical examples.

See the complete agenda for this course: debugging-online-agenda.pdf. The complete training materials will be freely available after the first session has been delivered, end of November 2022.

This course is currently being prepared by Bootlin engineer Clément Léger, who will also be teaching this course. Clément has a deep knowledge of how CPUs work and how Linux runs on a given CPU architecture, by having ported the Linux kernel to a brand new CPU architecture.

While the first session of this course will be delivered to a private customer, we have already scheduled a first public session which will take place on January 30, 31, Feb 1, 2, 2023, plus an extra session on Feb 3, 2023 if needed to cover all topics/questions. This session will take place each day from 14:00 to 18:00 UTC+1 (Paris time). Registration takes place directly online: the pricing is 569 EUR per seat at the discounted rate, or at 669 EUR per seat at the normal rate.

We can also organize a private session of this course for your team/company, either on-line or on-site: contact us to request a quote if you’re interested.

The videos from Bootlin’s presentations earlier this month at FOSDEM 2021 are now publicly available. Once again, FOSDEM was a busy event, even if it was online for once. As in most technical conferences, Bootlin engineers volunteered to share their experience and research by giving two talks.

Maxime Chevallier – Network Performance in the Linux Kernel, Getting the most out of the Hardware

Abstract: The networking stack is one of the most complex and optimized subsystems in the Linux kernel, and for a good reason. Between the wild range of applications, the complexity and variety of the networking hardware, getting good performance while keeping the stack easily usable from userspace has been a long-standing challenge.

Nowadays, complex Network Interface Controllers (NICs) can be found even on small embedded systems, bringing powerful features that were previously found only in the server world closest to day to day users.

This is a good opportunity to dive into the Linux Networking stack, to discover what is used to make networking as fast as possible, both by using all the features of the hardware and by implementing some clever software tricks.

In this talk, we cover these various techniques, ranging from simple batch processing with NAPI, queue management with RSS, RPS, XPS and so on, flow steering and filtering with ethool and TC, to finish with the newest big change that is XDP.

We dive into these various techniques and see how to configure them to squeeze the most out of your hardware, and discover that what was previously in the realm of datacenters and huge computers can now also be applied to embedded linux development.

Here are PDF slides for this presentation.

Michael Opdenacker – Embedded Linux from Scratch in 45 minutes, on RISC-V

Abstract: Discover how to build your own embedded Linux system completely from scratch. In this presentation and tutorial, we show how to build a custom toolchain (Buildroot), bootloader (opensbi / U-Boot) and kernel (Linux), that one can run on a system with the new RISV-V open Instruction Set Architecture emulated by QEMU. We also show how one can build a minimal root filesystem by oneself thanks to the BusyBox project. The presentation ends by showing how to control the system remotely through a tiny webserver. The approach is to provide only the files that are strictly necessary. That’s all the interest of embedded Linux: one can really control and understand everything that runs on the system, and see how simple the system can be. That’s much easier than trying to understand how a GNU/Linux system works from a distribution as complex as Debian!

The presentation also shares details about what’s specific to the RISC-V architecture, in particular about the various stages of the boot process. This presentation shares all the hardware (!), source code build instructions and demo binaries needed to reproduce everything at home, and add specific improvements. Most of the details are also useful to people using other hardware architectures (in particular arm and arm64).

It’s probably the first time a tutorial manages to show so many aspects of embedded Linux in less than an hour. See by yourself! At least, that’s for sure the first one demonstrating how to boot Linux from U-Boot in a RISC-V system emulated by QEMU.

Here are PDF slides for this presentation.