Poison, redzones and shadows: inside KASAN

Even if the presence of Rust is slowly increasing in the Linux kernel code base, the project largely remains written in C, and while this is the de facto language to write low level code, it unfortunately also comes with a significant ability of making mistakes, with the corresponding failures then coming in a wide variety of shapes: undefined behaviors, buffer overflows, segmentation faults… The kernel is not immune to such issues, and so kernel developers need some dedicated tooling to catch those issues early. As many problems have their roots in memory management, one tool that can legitimately sit in any kernel hacker toolbox is the Kernel Address Sanitizer, or KASAN for short. This blog does not aim to teach readers how to use KASAN: the kernel documentation is pretty explicit on this matter. We are rather going to explore KASAN internals, mostly to understand its impact (both at build time and at runtime), but up to some extent, to appreciate the elegant engineering involved in its implementation!

Continue reading “Poison, redzones and shadows: inside KASAN”

Debugging, tracing and profiling training updated with new eBPF content

Back in January 2023, we announced the availability of a new Bootlin training course  “Linux debugging, profiling, tracing and performance analysis”, initially developed and taught by former Bootlin engineer Clement Léger. A year and a half later, the training course, now taught by Bootlin engineers Luca Ceresoli and Alexis Lothoré, is one of the most requested amongst Bootlin training courses.

Following the feedback received from many people who attended the course, we are happy to announce that a new version of the training is available and  now includes a whole new part about eBPF technology !

Continue reading “Debugging, tracing and profiling training updated with new eBPF content”

New training course: Linux debugging, profiling, tracing and performance analysis

Since its inception, Bootlin has offered training courses on technical topics related to the use of Linux in embedded systems, with freely accessible training materials, and trainers with in-depth and real-life experience in their field. Based on these ideas, we have progressively extended our training portfolio over the years.

Today, we are extremely happy to announce a brand new course: Linux debugging, profiling, tracing and performance analysis. Many of our customers working on embedded Linux systems have expressed interest in diving into these complex topics, so we’ve created this course to:

  1. Give enough background information about how Linux works to be able to have a solid reasoning when investigating performance issues or simply bugs. Our course therefore details how user-space vs. kernel-space works, scheduling, and memory management, as a prerequisite to understanding better how Linux works.
  2. Give a strong introduction to the most important debugging, profiling and tracing tools in Linux, which are often not easy to get started with. We cover a very large spectrum of tools: strace, gdb, perf, ftrace, LTTng, kgdb, kmemleak, and many more. All these tools are illustrated through practical examples.

See the complete agenda for this course: debugging-online-agenda.pdf. The complete training materials will be freely available after the first session has been delivered, end of November 2022.

This course is currently being prepared by Bootlin engineer Clément Léger, who will also be teaching this course. Clément has a deep knowledge of how CPUs work and how Linux runs on a given CPU architecture, by having ported the Linux kernel to a brand new CPU architecture.

While the first session of this course will be delivered to a private customer, we have already scheduled a first public session which will take place on January 30, 31, Feb 1, 2, 2023, plus an extra session on Feb 3, 2023 if needed to cover all topics/questions. This session will take place each day from 14:00 to 18:00 UTC+1 (Paris time). Registration takes place directly online: the pricing is 569 EUR per seat at the discounted rate, or at 669 EUR per seat at the normal rate.

We can also organize a private session of this course for your team/company, either on-line or on-site: contact us to request a quote if you’re interested.

Android gdbclient command

Before you even start building Android, Google’s instructions tell you to source the build/envsetup.sh shell script.

This script exports a number of environment variables (that’s why you have to source it), mostly setting the PATH to your different toolchains and to your output directories.

It also defines a number of shell functions. Among them some functions are advertised, like the well-know lunch, that is used to configure to some extent the build system, or the grepping functions, but some are not, like pid, which uses adb to get the PID of a process running on the device.

Among the latter, one seems pretty useful: gdbclient. What gdbclient does is obviously gdb related but in fact it does more than that.

First, you run it by doing gdbclient <binary>:<port> <process_name>

Then it sets up adb with the forward command so that you use it as a transport layer to your device, while it appears as (in that case) opened TCP sockets both on your machine and on the device.

Then, it attaches a gdbserver to the process you gave as the third argument on the device.

Finally, it launches your cross-gdb on your workstation, loads the debugging symbols from the file passed as first argument, and sets up a remote debugging session. All of that through USB!

This is definitely useful, and I can’t say why Google doesn’t advertise it more, but hey, it’s there!