Linux kernel driver and Yocto training courses now on BeaglePlay

BeaglePlayLast summer, we announced the availability of our Embedded Linux course on the BeaglePlay platform.

Today, we are happy to announce the publication of our Linux kernel driver development and Yocto Project/OpenEmbedded system development courses also ported on the BeaglePlay platform.

Continue reading “Linux kernel driver and Yocto training courses now on BeaglePlay”

Debugging, tracing and profiling training updated with new eBPF content

Back in January 2023, we announced the availability of a new Bootlin training course  “Linux debugging, profiling, tracing and performance analysis”, initially developed and taught by former Bootlin engineer Clement Léger. A year and a half later, the training course, now taught by Bootlin engineers Luca Ceresoli and Alexis Lothoré, is one of the most requested amongst Bootlin training courses.

Following the feedback received from many people who attended the course, we are happy to announce that a new version of the training is available and  now includes a whole new part about eBPF technology !

Continue reading “Debugging, tracing and profiling training updated with new eBPF content”

Embedded Linux audio training course: materials published

Bootin Audio trainingEnd of June, we announced the availability of a brand new training course, Embedded Linux Audio, which is targeted at engineers working with audio on embedded Linux systems, and that covers topics ranging from audio hardware, audio support in the Linux kernel (ASoC, DAI and codec drivers, Device Tree representation), audio support in user-space (alsa-lib, alsa-utils, PipeWire, GStreamer).

We are pleased to announce today that the training materials are now available for download. This again shows our commitment to sharing all our training materials for free, under an open-source license (Creative Commons BY-SA).

This training course has already been given earlier this year to a private customer, and we are releasing the materials before the very first public session, which will take place on September 11 to September 14, and which will be taught by Bootlin engineer and COO Alexandre Belloni. If you’re interested, registration is open, we have a few seats remaining.

Bootlin Embedded Linux course now available on BeaglePlay

BeaglePlayEarlier this year, the BeagleBoard.org foundation announced the availability of a new development board, the BeaglePlay, centered around the Texas Instruments AM625 ARM64 processor. Bootlin has for a long time supported the BeagleBone Black as one of the hardware platforms used in the practical labs of its training courses, and we are happy to announce today that we now have added support for the BeaglePlay to our most popular course, the Embedded Linux system development course.

This means that participants to our Embedded Linux system development course can now choose to perform the practical labs on the BeaglePlay, experiencing a new and modern hardware platform, based on an ARM64 processor. This applies not only to participants choosing to purchase our course, but also to everyone else in the world, as our training materials are all freely available and distributed under an open-source license. The materials are all available on our Embedded Linux system development training course page:

We would like to thank Clément Ramirez who has worked with us on this effort of porting this course to the BeaglePlay platform.

“Bootlin is a leading provider of high-quality, in-depth Linux education. What amazes me is how much they not only support open source with outstanding teaching, but how much they stand behind the principles of open source by sharing their teaching materials and contributing to the open source projects that drive them.” said Jason Kridner, founder of BeagleBoard.org. “BeagleBoard.org shares these principles and that is why I believe BeaglePlay is an excellent choice. BeaglePlay doesn’t just run open source software, the hardware itself is open source with detailed documentation on the board and associated Texas Instruments AM625 system-on-chip, with availability to enable hobbyists and professionals alike from prototype to production. I couldn’t be more excited.”

Bootlin’s Embedded Linux course, like all our training courses, is available in 3 different options:

  • Public on-line sessions, delivered by video-conference, with a per-participant registration, and sessions organized at dates scheduled by Bootlin. See this page for more details, dates and registration process
  • Private on-line sessions, also delivered by video-conference, organized privately for your team, at the date of your choice. See this page for more details
  • Private on-site sessions, where one of our experienced engineers and trainers travels to your location, to teach your team in person. See this page for more details

In both our private on-line sessions and private on-site sessions, our customers are free to chose, among our set of supported platforms, the HW platform they would like to use for the practical labs, and this now includes the BeaglePlay. In our public on-line sessions, the trainer demonstrates the practical labs on one particular platform, but participants are able to reproduce the labs on the platform of their choice among our supported platforms.

With this new development, we look forward to continue our mission of helping the broader engineering community get trained on embedded Linux technology and expand the number of users and contributors to open-source technologies and communities.

Boot time: choose your kernel loading address carefully

When the compressed and uncompressed kernel images overlap

At least on ARM32, there seems to be many working addresses where the compressed kernel can be loaded in RAM. For example, one can load the compressed kernel at offset 0x1000000 (16 MB) from the start of RAM, and the Device Tree Blog (DTB) at offset 0x2000000 (32 MB). Whatever this loading address, the kernel is then decompressed at offset 0x8000 from the start of RAM, as explained this the famous How the ARM32 Linux kernel decompresses article from Linus Walleij.

There is a potential issue with the loading address of the compressed kernel, as explained in the article too. If the compressed kernel is loaded too close to the beginning of RAM, where the kernel must be decompressed, there will be an overlap between the two. The decompressed kernel will overwrite the compressed one, potentially breaking the decompression process.

Overlapping compressed and decompressed kernel

As you see in the above diagram, when this happens, the bootstrap code in the compressed kernel will first copy the compressed image to a location that’s far enough to guarantee that the decompressed kernel won’t overlap it. However, this extra step in the boot process has a cost.

Measuring boot time impact

In the context of updating our materials for our upcoming Embedded Linux Boot Time Optimization course in June, we measured this additional time on the STM32MP157A-DK1 Discovery Kit from STMicroelectronics, with a dual-core ARM Cortex-A7 CPU running at 650 MHz.

Initially, in our Embedded Linux System Development course, we were booting the DK1 board as follows:

ext4load mmc 0:4 0xc0000000 zImage; ext4load mmc 0:4 0xc4000000 dtb; bootz 0xc0000000 - 0xc4000000

0xc0000000 is exactly the beginning of RAM! We are therefore in the overlap situation.

We used grabserial from Tim Bird to measure the time between Starting kernel in U-Boot and when the compressed kernel starts executing (Booting Linux on physical CPU 0x0):

...
[4.451996 0.000124] Starting kernel ...
[0.001838 0.001838] 
[2.439980 2.438142] [    0.000000] Booting Linux on physical CPU 0x0
...

On a series of 5 identical tests, we obtained an average time of 2,440 ms, with a standard deviation of 0.4 ms.

Then, we measured the optimum case, in which the compressed kernel is loaded far enough from the beginning of RAM so that no overlap is possible:

No overlap between compressed and decompressed kernel

Here we chose to load the kernel at 0xc2000000:

ext4load mmc 0:4 0xc2000000 zImage; ext4load mmc 0:4 0xc4000000 dtb; bootz 0xc2000000 - 0xc4000000

On a series of 5 identical tests, we obtained an average time of 2,333 ms, with a standard deviation of 0.7 ms.

The new average is 107 ms smaller, which you are likely to consider as a worthy reduction, if you have experience with boot time reduction projects.

What to remember

In your embedded projects, if you are using a compressed kernel, make sure it is loaded far enough from the beginning of RAM, leaving enough space for the decompressed kernel to fit in between. Otherwise, your system will still be able to boot, but depending on the speed of your CPU and storage, it will be slower, from a few tens to a few hundreds of milliseconds.

We checked the How to optimize the boot time page on the STM32 wiki, and it recommends optimum loading addresses: 0xc2000000 for the kernel and 0xc4000000 for the device tree. This way, the upper limit for the decompressed kernel is 32 MB, which is more than enough.

If you are directly using an uncompressed kernel, which is more rare, you should also make sure that it is loaded at an optimum location, so that there is no need to move it before starting it.

Embedded Linux and Yocto training courses now available in Italian

Flag of ItalyBuongiorno!

For many years, we have been offering our training courses in both English and French. We are happy to announce that we are now expanding our offering towards our Italian customers: we are now able to deliver our very popular Embedded Linux system development and Yocto Project and OpenEmbedded development training courses in Italian. Those courses are delivered by a native Italian speaker, Bootlin engineer Luca Ceresoli. Luca has 20 years of experience working with embedded systems, including 15 years on embedded Linux.

Thanks to this offering, our Italian customers can enjoy a training course delivered in their native language, which obviously facilitates communication, understanding and interaction. We can offer our Embedded Linux course and Yocto course in Italian through either private on-site sessions or private on-line sessions. Contact us at training@bootlin.com for details and pricing!

Ti aspettiamo!

Embedded Linux course labs now available on Beagle Bone

Following the release of the improved version of our embedded Linux course, we are happy to announce that we have finished porting our new labs to the BeagleBone Black and BeagleBone Black wireless boards.

Continue reading “Embedded Linux course labs now available on Beagle Bone”

New training course: Linux debugging, profiling, tracing and performance analysis

Since its inception, Bootlin has offered training courses on technical topics related to the use of Linux in embedded systems, with freely accessible training materials, and trainers with in-depth and real-life experience in their field. Based on these ideas, we have progressively extended our training portfolio over the years.

Today, we are extremely happy to announce a brand new course: Linux debugging, profiling, tracing and performance analysis. Many of our customers working on embedded Linux systems have expressed interest in diving into these complex topics, so we’ve created this course to:

  1. Give enough background information about how Linux works to be able to have a solid reasoning when investigating performance issues or simply bugs. Our course therefore details how user-space vs. kernel-space works, scheduling, and memory management, as a prerequisite to understanding better how Linux works.
  2. Give a strong introduction to the most important debugging, profiling and tracing tools in Linux, which are often not easy to get started with. We cover a very large spectrum of tools: strace, gdb, perf, ftrace, LTTng, kgdb, kmemleak, and many more. All these tools are illustrated through practical examples.

See the complete agenda for this course: debugging-online-agenda.pdf. The complete training materials will be freely available after the first session has been delivered, end of November 2022.

This course is currently being prepared by Bootlin engineer Clément Léger, who will also be teaching this course. Clément has a deep knowledge of how CPUs work and how Linux runs on a given CPU architecture, by having ported the Linux kernel to a brand new CPU architecture.

While the first session of this course will be delivered to a private customer, we have already scheduled a first public session which will take place on January 30, 31, Feb 1, 2, 2023, plus an extra session on Feb 3, 2023 if needed to cover all topics/questions. This session will take place each day from 14:00 to 18:00 UTC+1 (Paris time). Registration takes place directly online: the pricing is 569 EUR per seat at the discounted rate, or at 669 EUR per seat at the normal rate.

We can also organize a private session of this course for your team/company, either on-line or on-site: contact us to request a quote if you’re interested.

Improved version of our popular “Embedded Linux system development” course

Embedded Linux system development courseOur Embedded Linux system development course has been for many years one of our most popular training courses. It is our course targeted at engineers who are getting started with Linux on embedded systems, and need to understand the big picture, but with a sufficiently deep level of details. It describes the overall structure of an embedded Linux system, and teaches step by step how it is build: cross-compilation toolchain, bootloader, Linux kernel, minimal root filesystem, storage, integration of user-space components, build systems, etc.

Even though this course has seen many small updates throughout the years to use newer versions of the different software components and various improvements and updates to the training materials, the course had remained fundamentally the same for quite some time. However, we had identified a number of areas on which we wanted to make some more fundamental changes, and we’ve taken the chance of the summer season to prepare a brand new version of this course, with many major changes and improvements.

This updated version of the course is now available, and as usual with Bootlin its training materials are freely available:

Embedded Linux system development courseCompared to our previous version of this course, the main changes are:

  • Major rewrite of the lecture section on bootloader and firmware, to better cover UEFI, Trusted Firmware (TF-A), Trusted Execution Environment, and overall reflect the increased complexity of the booting process of modern embedded platforms. The corresponding practical lab also makes use of TF-A to illustrate this.
  • Addition of a new section on Accessing hardware devices with many details on the Device Tree, how to identify kernel drivers for devices, and what are the typical interfaces in Linux to access hardware. This is illustrated by a new lab in which we manipulate GPIOs, LEDs, add support for a sound card connected over USB, and add support for a joystick connected over I2C, which extending the Device Tree and manipulating pin-muxing.
  • Removal of the practical lab on flash filesystems, due to the progressively reducing number of platforms that use raw NAND flash. We still have a lecture on how raw flash memory is handled in Linux (MTD, UBI, UBIFS), but no longer a practical lab, in order to spend time on topics that are more commonly relevant.
  • A major rework of the final part of the course which covers the user-space stack, with the aim of showing how to build a reasonably realistic product:
    • We explain and demonstrate how to cross-compile and integrate manually libraries and applications in an embedded Linux system. We illustrate this by cross-compiling alsa-lib and alsa-utils to play audio with our USB audio device, libgpiod to manipulate GPIOs, and ipcalc to manipulate the Meson build system.
    • We then explain the concepts and principles behind embedded Linux build systems (with a focus on Buildroot and Yocto), as well as binary distributions. We illustrate this by using Buildroot to build a complete embedded Linux system, which uses mpd as an audio player daemon.
    • We then cover open-source licensing topics.
    • We then cover the major Linux software stacks for graphics, multimedia, networking, as well as systemd and D-Bus. We illustrate this by changing our system to use systemd as an init system, and use udev for device management.
    • Finally, we cover application development and debugging: how to cross-compile your own application, how to debug it, using strace, ltrace, gdb, perf, valgrind. We illustrate this by implementing an application that allows to use an I2C-connected joystick to control the audio playback of an audio playlist. This application is then analyzed and debugged using the relevant debugging tools.
  • The real-time Linux part of the course has been removed from the course, as we now have a dedicated real-time Linux with PREEMPT_RT course, which goes into many more details on this topic.

The course is currently delivered with practical labs done on the STM32MP1 platform from ST, but we intend to port it on the BeagleBone Black and Qemu as well. In any case, the course is very generic and relevant for all embedded Linux projects, regardless of the specific hardware platform being used.

This new edition has already been delivered to several customers, both on-line and on-site. All our slots for 2022 are already fully booked, but we do offer for 2023:

  • A public on-line session on January 30, 31, February 2, 3, 6, 7, 9, 2023, from 2:00 PM to 6:00 PM UTC+1. Registration is accessible directly online. This is the best option to train a few engineers from your team.
  • Private sessions, either on-line or on-site, upon request. You can contact us to discuss the details and get a quote.

We are really looking forward to continuing to share our knowledge about embedded Linux with even more engineers, and are confident that this updated version of the course will make this knowledge sharing even more efficient and fruitful.

Yocto training course updated to Kirkstone release

Yocto ProjectBack in May 2022, the Yocto Project published the Kirkstone release, the latest Long Term Support version of the popular embedded Linux build system. See the release notes of this 4.0 release.

For many years, Bootlin has helped plenty of engineers around the world get started with the Yocto Project and OpenEmbedded thanks to our Yocto Project and OpenEmbedded development training course, whose training materials are freely accessible, to everyone.

We are happy to announce that we have just published the update of these materials to cover this new Kirstone release. Our practical lab instructions, available for both the STM32MP1 Discovery Kit and the BeagleBoneBlack, have been correspondingly updated:

This update was done by Bootlin engineer Luca Ceresoli, who is teaching this updated version of our course this week. The next public session, open to individual registration, will take place on September 26-30, and will be taught by Maxime Chevallier. If you are interested, you can register directly online. We also offer this course in private sessions, organized on demand for your team, either on-line or on-site, you can contact us for more details.