Improved version of our popular “Embedded Linux system development” course

Embedded Linux system development courseOur Embedded Linux system development course has been for many years one of our most popular training courses. It is our course targeted at engineers who are getting started with Linux on embedded systems, and need to understand the big picture, but with a sufficiently deep level of details. It describes the overall structure of an embedded Linux system, and teaches step by step how it is build: cross-compilation toolchain, bootloader, Linux kernel, minimal root filesystem, storage, integration of user-space components, build systems, etc.

Even though this course has seen many small updates throughout the years to use newer versions of the different software components and various improvements and updates to the training materials, the course had remained fundamentally the same for quite some time. However, we had identified a number of areas on which we wanted to make some more fundamental changes, and we’ve taken the chance of the summer season to prepare a brand new version of this course, with many major changes and improvements.

This updated version of the course is now available, and as usual with Bootlin its training materials are freely available:

Embedded Linux system development courseCompared to our previous version of this course, the main changes are:

  • Major rewrite of the lecture section on bootloader and firmware, to better cover UEFI, Trusted Firmware (TF-A), Trusted Execution Environment, and overall reflect the increased complexity of the booting process of modern embedded platforms. The corresponding practical lab also makes use of TF-A to illustrate this.
  • Addition of a new section on Accessing hardware devices with many details on the Device Tree, how to identify kernel drivers for devices, and what are the typical interfaces in Linux to access hardware. This is illustrated by a new lab in which we manipulate GPIOs, LEDs, add support for a sound card connected over USB, and add support for a joystick connected over I2C, which extending the Device Tree and manipulating pin-muxing.
  • Removal of the practical lab on flash filesystems, due to the progressively reducing number of platforms that use raw NAND flash. We still have a lecture on how raw flash memory is handled in Linux (MTD, UBI, UBIFS), but no longer a practical lab, in order to spend time on topics that are more commonly relevant.
  • A major rework of the final part of the course which covers the user-space stack, with the aim of showing how to build a reasonably realistic product:
    • We explain and demonstrate how to cross-compile and integrate manually libraries and applications in an embedded Linux system. We illustrate this by cross-compiling alsa-lib and alsa-utils to play audio with our USB audio device, libgpiod to manipulate GPIOs, and ipcalc to manipulate the Meson build system.
    • We then explain the concepts and principles behind embedded Linux build systems (with a focus on Buildroot and Yocto), as well as binary distributions. We illustrate this by using Buildroot to build a complete embedded Linux system, which uses mpd as an audio player daemon.
    • We then cover open-source licensing topics.
    • We then cover the major Linux software stacks for graphics, multimedia, networking, as well as systemd and D-Bus. We illustrate this by changing our system to use systemd as an init system, and use udev for device management.
    • Finally, we cover application development and debugging: how to cross-compile your own application, how to debug it, using strace, ltrace, gdb, perf, valgrind. We illustrate this by implementing an application that allows to use an I2C-connected joystick to control the audio playback of an audio playlist. This application is then analyzed and debugged using the relevant debugging tools.
  • The real-time Linux part of the course has been removed from the course, as we now have a dedicated real-time Linux with PREEMPT_RT course, which goes into many more details on this topic.

The course is currently delivered with practical labs done on the STM32MP1 platform from ST, but we intend to port it on the BeagleBone Black and Qemu as well. In any case, the course is very generic and relevant for all embedded Linux projects, regardless of the specific hardware platform being used.

This new edition has already been delivered to several customers, both on-line and on-site. All our slots for 2022 are already fully booked, but we do offer for 2023:

  • A public on-line session on January 30, 31, February 2, 3, 6, 7, 9, 2023, from 2:00 PM to 6:00 PM UTC+1. Registration is accessible directly online. This is the best option to train a few engineers from your team.
  • Private sessions, either on-line or on-site, upon request. You can contact us to discuss the details and get a quote.

We are really looking forward to continuing to share our knowledge about embedded Linux with even more engineers, and are confident that this updated version of the course will make this knowledge sharing even more efficient and fruitful.

Bootlin toolchains 2022.08 released

Bootlin toolchains 2022.08The toolchains.bootlin.com service provides freely available pre-compiled cross-compilation toolchains for a wide range of CPU architectures and configurations.

We have just published version 2022.08 of these toolchains, which are now built using Buildroot 2022.08. Thanks to this the toolchains now use the following components:

  • The bleeding-edge toolchains are based on gcc 12, binutils 2.39, gdb 12, kernel headers 5.4, glibc 2.35 or musl 1.2.3 or uclibc-ng 1.0.42.
  • The stable toolchains are based on gcc 11, binutils 2.38, gdb 11, kernel headers 4.9, glibc 2.35 or musl 1.2.3 or uclibc-ng 1.0.42.

Even though glibc 2.36 has been recently released, it still caused too many issues to be integrated in this toolchain release. glibc 2.36 will be part of a future update of these toolchains.

We also have a few new toolchains that appeared:

  • Toolchains for the OpenRISC CPU architecture, based on the glibc C library. We already had toolchains for OpenRISC, but not using glibc.
  • We now have both stable and bleeding-edge toolchains for the x86-64-v2, x86-64-v3 and x86-64-v4 architecture variants. We used to have only bleeding-edge toolchains for these variants as only the latest gcc had support for them.

The only toolchain that was not updated as part of this release is the m68k Coldfire toolchain, as we currently have a regression on elf2flt. This will hopefully be adressed in the future.

If you have feedback or encounter any issue in using these toolchains, the project issue tracker is where you should go.

Welcome to Théo Lebrun!

Welcome on board!We are happy to announce that Théo Lebrun has joined our engineering team, as of September 9, 2022, just in time to attend the Embedded Linux Conference Europe with the rest of the Bootlin engineering team.

Théo has just graduated from the Université de Technologie de Belfort Montbéliard, and did his final 6-month internship at Bootlin, where he worked on PipeWire and support for the A2B Audio Bus in the Linux kernel.

Théo is joining our team based in Lyon, where Alexandre Belloni, Grégory Clement and Kamel Bouhara are based.

Welcome on board Théo!

A journey in the RTC subsystem

As part of a team effort to improve the upstream Linux kernel support for the Renesas RZ/N1 ARM processor, we had to write from scratch a new RTC driver for this SoC. The RTC subsystem API is rather straightforward but, as most kernel subsystems, the documentation about it is rather sparse. So what are the steps to write a basic RTC driver? Here are some pointers.

The registration

The core expects drivers to allocate, initialize and then register a struct rtc_device with the device managed helpers: devm_rtc_allocate_device() and devm_rtc_register_device(). Between these two function calls, one will be required to provide at least a set of struct rtc_class_ops which contains the various callbacks used to access the device from the core, as well as setting a few information about the device.

The kind of information expected is the support for various features (rtcdev->features bitmap) as well as the maximum continuous time range supported by your RTC. If you do not know the actual date after which your device stops being reliable, you can use the rtc-range test tool from rtc-tools, available at https://git.kernel.org/pub/scm/linux/kernel/git/abelloni/rtc-tools.git (also available as a Buildroot package). It will check the consistency of your driver against a number of common known-to-be-failing situations.

Time handling

The most basic operations to provide are ->read_time() and ->set_time(). Both functions should play with a struct rtc_time which describes time and date with members for the year, month, day of the month, hours (in 24-hour mode), minutes and seconds. The week day member is ignored by userspace and is not expected to be set properly, unless it is actively used by the RTC, for example to set alarms. There are then three popular ways of storing time in the RTC world:

  1. either using the binary values of each of these fields
  2. or using a Binary Coded Decimal (BCD) version of these fields
  3. or, finally, by storing a timestamp in seconds since the epoch

In BCD, each decimal digit is encoded using four bits, eg. the number 12 could either be coded by 0x0C in hexadecimal, or 0x12 in BCD, which is easier to read with a human eye.

The three representations are absolutely equivalent and you are free to convert the time from one system to another when needed:

  • #1 <-> #2 conversions are done with bcd2bin() and bin2bcd() (from linux/bcd.h)
  • #1 <-> #3 conversions are done with rtc_time64_to_tm() and rtc_tm_to_time64() (from linux/rtc.h)

While debugging, it is likely that you will end up dumping these time structures. Note that struct rtc_time is aligned on struct tm, this means that the year field is the number of years since 1900 and the month field is the number of months since January, in the range 0 to 11. Anyway, dumping these fields manually is a loss of time, it is advised instead to use the dedicated RTC printk specifiers which will handle the conversion for you: %ptR for a struct rtc_time, %ptT for a time64_t.

Of course, when reading the actual time from multiple registers on the device and filling those fields, be aware that you should handle possible wrapping situations. Either the device has an internal latching mechanism for that (eg. the front-end of the registers that you must read are all frozen upon a specific action) or you need to verify this manually by, for instance, monitoring the seconds register and try another read if it changed between the beginning and the end of the retrieval.

If your device continuous time range ended before 2000 you may want to shift the default hardware range further by providing the start-year device tree property. The core will then shift the Epoch further for you.

Finally, once done, you can verify your implementation by playing with the rtc test tool (also from rtc-tools).

Supporting alarms

One common RTC feature is the ability to trigger alarms at specific times. Of course it’s even better if your RTC can wake-up the system.

If the device or the way it is integrated doesn’t support alarms, this should be advertised at registration time by clearing the relevant bit (RTC_FEATURE_ALARM, RTC_FEATURE_UPDATE_INTERRUPT). In the other situations, it is relevant to indicate whether the RTC has a second, 2-seconds or minute resolution by setting the appropriate flag (RTC_FEATURE_ALARM_RES_2S, RTC_FEATURE_ALARM_RES_MINUTE). Mind when testing that querying an alarm time below this resolution will return a -ETIME error.

When implementing the ->read_alarm(), ->set_alarm() and ->alarm_irq_enable() hooks, be aware that the update and periodic alarms are now implemented in the core, using HR timers rather than with the RTC so you should focus on the regular alarm. The read/set hooks naturally allow to read and change the alarm settings. A struct rtc_wkalrm *alrm is passed as parameter, alrm->time is the struct rtc_time and alrm->enabled the state of the alarm (which must be set in ->set_alarm()). The third hook is an asynchronous way to enable/disable the alarm IRQ.

The interrupt handler for the alarm is required to call rtc_update_irq() to singal the core that an alarm happened, providing the RTC device, the number of alarms reported (usually one), and the RTC_IRQF flag OR’ed with the relevant alarm flag (likely, RTC_AF for the main alarm).

Oscillator offset compensation

RTC counters rely on very precise clock sources to deliver accurate times. When the source is not matching the expected precision, which is the case of most cheap oscillators on the market. Some RTCs have a mechanism allowing to compensate for the frequency variation by incrementing or skipping the RTC counters at a regular interval in order to get closer to the reality.

The RTC subsystem offers a set of callbacks, ->read_offset() and a ->set_offset(), where a signed offset is passed in ppb (parts per billion).

As an example, if an oscillator is below its targeted frequency of 32768 Hz and is measured to run at 32767.7 Hz, we need to offset the counter by 1 - (32767.7/32768) = 9155 ppb. If the RTC is capable of offsetting the main counter once every 20s it means that every 20s, this counter (which gets decremented at the frequency of the oscillator to produce the “seconds”) will start at a different value than 32768. Adding 1 to this counter every 20s would basically mean earning 1 / (32768 * 20) = 1526 ppb. Our target being 9155 ppb, we must offset the counter by 9155 / 1526 = 6 every 20s to get a compensated rate of 32767.7 + (6 / 20) = 32768 Hz.

Upstreaming status of the RZ/N1 RTC driver

The RZ/N1 RTC driver has all the features listed above and made its way into the v5.18 Linux kernel release. Hopefully this little reference sheet will encourage others to finalize and send new RTC drivers upstream!

Bootlin at Linux Plumbers conference 2022

Next week, almost the entire Bootlin team will be at the Embedded Linux Conference Europe in Dublin, see our previous blog post on this topic. We will give four talks at this event, on a variety of Linux kernel and embedded Linux topics.

During the same week, also in Dublin albeit in a different location, will take place the Linux Plumbers conference. Bootlin engineer Miquèl Raynal will give a talk at Linux Plumbers, as part of the IoTs a 4-Letter Word micro-conference. Miquèl’s talk will discuss Linux IEEE 802.15.4 MLME improvements, as Miquèl has been working for several months on bringing improvements to the 802.15.4 stack in the Linux kernel.