Videos and slides of Bootlin’s talks at Live Embedded Event 2020

Yesterday, Bootlin co-organized and participated to the first edition of Live Embedded Event, a new online conference dedicated to embedded systems topics. In addition to co-organizing the event, we also gave four different talks at this conference, and we are happy to share the slides and videos of our talks.

Introduction video

The following video was presented during the introduction keynote of the event, where each sponsor and co-organizer could say a few words.

An overview of the Linux kernel and user-space graphics stack

Talk given by Bootlin engineer Paul Kocialkowski, slides available in PDF.

ASoC: Supporting Audio on an Embedded Board

Talk given by Bootlin engineer Alexandre Belloni, slides available in PDF.

Embedded Linux “from scratch” in 45 minutes…on RISC-V

Talk given by Bootlin founder and CEO Michael Opdenacker, slides available in PDF.

Device Tree: hardware description for everybody!

Talk given by Bootlin CTO Thomas Petazzoni, slides available in PDF.

4 talks from Bootlin at Live Embedded Event, December 3

As we announced back in October, Bootlin has participated to the organization of a new online conference around embedded systems: Live Embedded Event, which will take place on December 3. The registration is totally free, and the event will propose 4 tracks throughout the day, covering a wide range of topics. We encourage you to register and participate to the event!

Bootlin is not only co-organizing the event, but also has 4 talks that have been accepted:

  • An overview of the Linux and user-space graphics stack, by Paul Kocialkowski, Bootlin’s multimedia expert and author of our Linux Graphics training course. Talk abstract: Graphics with the Linux kernel is often perceived as a haystack, composed of many components that have complex interactions with eachother. While this is not always far from the truth, this talk will try to provide a clear overview of the different pieces in the stack and what their individual role is. Throughout the presentation, various APIs, protocols and software implementations will be demystified, covering both display and render aspects. This includes topics such as DRM, Wayland, Xorg, OpenGL, Mesa and more!
  • Embedded Linux from scratch in 45 minutes… on RISC-V, by Michael Opdenacker, Bootlin’s founder and CEO. Talk abstract: Join and discover how to build your own embedded Linux system completely from scratch. You will build your own toolchain, bootloader and kernel, this time on the Risc-V open Instruction Set Architecture, that you will run through a system emulated by QEMU. You will also build a minimal root filesystem by yourself thanks to the BusyBox project. You will finish by controlling the system through a tiny webserver. The approach will be to provide only the files that are strictly necessary. That’s all the interest of embedded Linux: you can really control and understand everything that runs on your system, and see how simple the system can be. That’s much easier than trying to understand how a GNU/Linux system works from a distribution as complex as Debian!
  • ASoC: audio on an embedded board, by Alexandre Belloni, Bootlin’s audio expert, Linux kernel and Yocto trainer, as well as kernel maintainer. Talk abstract: ASoC, which stands for ALSA System on Chip, is a Linux kernel subsystem created to provide better ALSA support for system-on-chip and portable audio codecs. It allows to reuse codec drivers across multiple architectures and provides an API to integrate them with the SoC audio interface. This talk will present the typical hardware architecture of audio devices on embedded platforms, how to integrate the audio codec and processor audio interface using device tree. Examples, common issues and debugging tips will also be discussed.
  • Device Tree: hardware description for everybody!, by Thomas Petazzoni, Bootlin’s CTO and co-maintainer of Buildroot. Talk abstract: The Device Tree has been adopted for the ARM 32-bit Linux kernel support almost a decade ago, and since then, its usage has expanded to many other CPU architectures in Linux, as well as bootloaders such as U-Boot and Barebox. Even though Device Tree is no longer a new thing, developers coming into the embedded Linux world often struggle to understand what Device Trees are, what is their syntax, how they interact with the Linux kernel device drivers, what Device Tree bindings are, and more. This talk will provide an introduction to the Device Tree, to jump start new developers in using this description language that is now ubiquitous in the vast majority of embedded Linux projects.

Note that we are aware of limitations of the platform that was chosen for the online conference: the details of the talks, and details about sponsors are only visible for people who registered. This is unfortunate, but we discovered this when it was too late to change the platform, together with a few other limitations. Let’s say this is a first edition, and regardless of those limitations, the selection of talks and contents is very promising!

Videos and slides from Bootlin talks at Embedded Linux Conference Europe 2020

The Embedded Linux Conference Europe took place online last week. While we definitely missed the experience of an in-person event, we strongly participated to this conference with no less than 7 talks on various topics showing Bootlin expertise in different fields: Linux kernel development in networking, multimedia and storage, but also build systems and tooling. We’re happy to be publishing now the slides and videos of our talks.

From the camera sensor to the user: the journey of a video frame, Maxime Chevallier

Download the slides: PDF, source.

OpenEmbedded and Yocto Project best practices, Alexandre Belloni

Download the slides: PDF, source.

Supporting hardware-accelerated video encoding with mainline Linux, Paul Kocialkowski

Download the slides: PDF, source.

Building embedded Debian/Ubuntu systems with ELBE, Köry Maincent

Download the slides: PDF, source.

Understand ECC support for NAND flash devices in Linux, Miquèl Raynal

Download the slides: PDF, source.

Using Visual Studio Code for Embedded Linux Development, Michael Opdenacker

Download the slides: PDF, source.

Precision Time Protocol (PTP) and packet timestamping in Linux, Antoine Ténart

Download the slides: PDF, source.

Enabling new hardware on Raspberry Pi with Device Tree Overlays

We recently had the chance to work on a customer project that involved the RaspberryPi Compute Module 3, with custom peripherals attached: a Microchip WILC1000 WiFi chip connected on SDIO, and a SGTL5000 audio codec connected over I2S/I2C. We take this opportunity to share some insights on how to introduce new hardware support on RaspberryPi platforms, by taking advantage of the Raspberry Pi specific Device Tree overlay mechanism.

Building and loading an upstream RPi Linux

Our customer is using the RaspberryPi OS Lite, which is a Debian-based system, coming with its pre-compiled Linux kernel image. However, for this project, we obviously need to customize the Linux kernel configuration, so we had to build our own kernel.

While Bootlin normally has a preference for using the upstream Linux kernel, for this project, we decided to keep using the Linux kernel fork provided by Raspberry Pi on Github, which is used by Raspberry Pi OS Lite. So we start our work by grabbing the Linux kernel source code from a the rpi-5.4.y provided by the Raspberry Pi Linux kernel repository:

$ git clone https://github.com/raspberrypi/linux -b rpi-5.4.y

After installing an appropriate ARM 32-bit cross-compilation toolchain, we’re ready to configure and build the kernel:

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j4 zImage

Thanks to the USB mass storage mode of the Raspberry Pi it is quite straightforward to update the kernel as it allows to mount the board eMMC partitions on the host system, and update whichever files we want. The tool that actually sets the RaspberryPi in this mode is called rpiboot.

After executing it, we deploy the zImage into the RaspberryPi /media/${USER}/boot/ partition and update the file /media/${USER}/boot/config.txt to set the firmware directive kernel=zImage.

That’s all it takes to replace the kernel image provided by the RaspberryPi OS Lite system by our own kernel, which means we are now ready to integrate our new features.

Device Tree Overlays

A specificity of the Raspberry Pi is that the boot flow starts from the GPU core and not the ARM core like it does on most embedded processors. The GPU load a first bootloader from a ROM that will load a second bootloader (bootcode.bin) from eMMC/SD card that is in charge of executing a firmware (start.elf). This GPU firmware finally reads and parses a file stored in the boot partition (config.txt), which is used to set various boot parameters such as the image to boot on.

This config.txt file also allows to indicate which Device Tree file should be used as the hardware description, as well as Device Tree Overlays that should be applied on top of the Device Tree files. Device Tree Overlays are a bit like patches for the Device Tree: they allow to extend the base Device Tree with new properties and nodes. They are typically used to describe the hardware attached to the RaspberryPi through expansion boards.

The Raspberry Pi kernel tree contains a number of such Device Tree Overlays in the arch/arm/boot/dts/overlays folder. Each of those overlays, stored in .dts file gets compiled into a .dtbo files. Those .dtbo can be loaded and applied to the main Device Tree by adding the following statement to the config.txt file:

dtoverlay=overlay-name,overlay-arguments

We will fully take advantage of this mechanism to introduce two new Device Tree Overlays that will be parsed and dynamically merged into the main Device Tree.

WILC1000 WiFi chip Device Tree overlay

The ATWILC1000 we want to support is a Microchip WiFi module. As of the Linux 5.4 kernel we are using for this project, the driver for this WiFi chip was still in the staging area of the Linux kernel, in drivers/staging/wilc1000, but it more recently graduated out of staging. We obviously started by enabling this driver in the kernel configuration, through its CONFIG_WILC1000 and CONFIG_WILC1000_SDIO options.

To describe the WILC1000 module in terms of Device Tree, we need to create a arch/arm/boot/dts/overlays/wilc1000-overlays.dts file, with the following contents:

/dts-v1/;
/plugin/;

/ {
    compatible = "brcm,bcm2835";

    fragment@0 {
        target = <&mmc>;
        wifi_ovl: __overlay__ {
            pinctrl-0 = <&sdio_ovl_pins &wilc_sdio_pins>;
            pinctrl-names = "default";
            non-removable;
            bus-width = <4>;
            status = "okay";
            #address-cells = <1>;
            #size-cells = <0>;

            wilc_sdio: wilc_sdio@0 {
                compatible = "microchip,wilc1000", "microchip,wilc3000";
                irq-gpios = <&gpio 1 0>;
                status = "okay";
                reg = <0>;
                bus-width = <4>;
            };
        };
    };

    fragment@1 {
        target = <&gpio>;
        __overlay__ {
            sdio_ovl_pins: sdio_ovl_pins {
                brcm,pins = <22 23 24 25 26 27>;
                brcm,function = <7>; /* ALT3 = SD1 */
                brcm,pull = <0 2 2 2 2 2>;
            };

            wilc_sdio_pins: wilc_sdio_pins {
                brcm,pins = <1>; /* CHIP-IRQ */
                brcm,function = <0>;
                brcm,pull = <2>;
            };
        };
    };
};

To be compiled, this overlay needs to be referenced in arch/arm/boot/dts/overlays/Makefile.

This overlay contains two fragments: fragment@0 and fragment@1. Each fragment will extend a Device Tree node of the main Device Tree, which is specified by the target property of each fragment: fragment@0 will extend the node referenced by the &mmc phandle, and fragment@1 will extend the node referenced by the &gpio phandle. The mmc and gpio labels are defined in the main Device Tree describing the system-on-chip, arch/arm/boot/dts/bcm283x.dtsi. Practically speaking, the first fragment enables the MMC controller, its pin-muxing, and describes the WiFi chip as a sub-node of the MMC controller. The second fragment describes the pin-muxing configurations used for the MMC controller.

Above, each fragment target, &mmc and &gpio is matching an existing device node in the underlying tree arch/arm/boot/dts/bcm283x.dtsi. Note that our hardware is designed such as the reset pin of the wilc is automatically de-assert when powering the board so we don’t define it here.

Now that we have our overlay we enable it through the firmware config.txt :

dtoverlay=sdio,poll_once=off
dtoverlay=wilc1000
gpio=38=op,dh

In addition to our own wilc1000 overlay, we are also loading the sdio overlay, with the poll_once=off argument to make sure we are polling our wilc device even after the boot is complete when the chip enable gpio is asserted through the firmware directive gpio=38=op,dh.

After copying the wilc1000.dtbo on our board we can verify that both overlays are getting merged in the main device-tree using the vcdbg command:

  pi@raspberrypi:~$ sudo vcdbg log msg
   002351.555: brfs: File read: /mfs/sd/config.txt
   002354.107: brfs: File read: 4322 bytes
   ...
   005603.860: dtdebug: Opened overlay file 'overlays/wilc1000.dtbo'
   005605.500: brfs: File read: /mfs/sd/overlays/wilc1000.dtbo
   005619.506: Loaded overlay 'sdio'
   005624.063: dtdebug: fragment 3 disabled
   005624.160: dtdebug: fragment 4 disabled
   005624.256: dtdebug: fragment 5 disabled
   005633.283: dtdebug: merge_fragment(/soc/mmcnr@7e300000,/fragment@0/__overlay__)
   005633.308: dtdebug:   +prop(status)
   005633.757: dtdebug: merge_fragment() end
   005642.521: dtdebug: merge_fragment(/soc/mmc@7e300000,/fragment@1/__overlay__)
   005642.549: dtdebug:   +prop(pinctrl-0)
   005643.003: dtdebug:   +prop(pinctrl-names)
   005643.443: dtdebug:   +prop(non-removable)
   005644.022: dtdebug:   +prop(bus-width)
   005644.477: dtdebug:   +prop(status)
   005644.935: dtdebug: merge_fragment() end
   005646.614: dtdebug: merge_fragment(/soc/gpio@7e200000,/fragment@2/__overlay__)
   005650.499: dtdebug: merge_fragment(/soc/gpio@7e200000/sdio_ovl_pins,/fragment@22
   /__overlay__/sdio_ovl_pins)
   ...

Of course there are lots of other useful tools that we used to debug, such as raspi-gpio and dtoverlay:

 pi@raspberrypi:~$ sudo raspi-gpio get
   BANK0 (GPIO 0 to 27):
   GPIO 0: level=1 fsel=0 func=INPUT
   GPIO 1: level=1 fsel=0 func=INPUT
   GPIO 2: level=1 fsel=0 func=INPUT
   ...
   GPIO 22: level=0 fsel=7 alt=3 func=SD1_CLK
   GPIO 23: level=1 fsel=7 alt=3 func=SD1_CMD
   GPIO 24: level=1 fsel=7 alt=3 func=SD1_DAT0
   GPIO 25: level=1 fsel=7 alt=3 func=SD1_DAT1
   GPIO 26: level=1 fsel=7 alt=3 func=SD1_DAT2
   GPIO 27: level=1 fsel=7 alt=3 func=SD1_DAT3

Here we can actually see that the correct pinmuxing configuration is set for the SDIO pins.

Testing WiFi on RaspberryPi OS

During our test we found that the wilc1000 seems to only support one mode at time which means only STA or soft-AP. The first mode is quite straightforward to test using the raspi-config tool, we just had have to supply our SSID name and password.

To know which modes are supported and if they are supported simultaneously use iw list:

   pi@raspberrypi:~$ iw list
   Wiphy phy0
           max # scan SSIDs: 10
           max scan IEs length: 1000 bytes
           max # sched scan SSIDs: 0
           max # match sets: 0
           max # scan plans: 1
           max scan plan interval: -1
           max scan plan iterations: 0
           Retry short limit: 7
           Retry long limit: 4
           Coverage class: 0 (up to 0m)
           Supported Ciphers:
                   * WEP40 (00-0f-ac:1)
                   * WEP104 (00-0f-ac:5)
                   * TKIP (00-0f-ac:2)
                   * CCMP-128 (00-0f-ac:4)
                   * CMAC (00-0f-ac:6)
           Available Antennas: TX 0x3 RX 0x3
           Supported interface modes:
                    * managed
                    * AP
                    * monitor
                    * P2P-client
                    * P2P-GO
            ...
        software interface modes (can always be added):
        interface combinations are not supported
        Device supports scan flush.
        Supported extended features:

To use the wilc1000 in soft-AP mode, one need to install additional packages in the RaspberryPi OS, such as hostapd and dnsmasq.

SGTL5000 audio codec Device Tree Overlay

After integrating support for the WILC1000 WiFi chip, we can now look at how we got the SGTL5000 audio codec to work. We wrote an overlay arch/arm/boot/dts/overlays/sgtl5000-overlay.dts with the following contents, and also edited arch/arm/boot/dts/overlays/Makefile to ensure it is built. Our sgtl5000-overlay.dts contains:

/dts-v1/;
/plugin/;

/{
    compatible = "brcm,bcm2835";

    fragment@0 {
        target-path = "/";
        __overlay__ {
            sgtl5000_mclk: sgtl5000_mclk {
                compatible = "fixed-clock";
                #clock-cells = <0>;
                clock-frequency = <12288000>;
                clock-output-names = "sgtl5000-mclk";
            };
        };
    };

    fragment@1 {
        target = <&soc>;
        __overlay__ {
            reg_1v8: reg_1v8@0 {
                compatible = "regulator-fixed";
                regulator-name = "1V8";
                regulator-min-microvolt = <1800000>;
                regulator-max-microvolt = <1800000>;
                regulator-always-on;
            };
        };
    };

    fragment@2 {
        target = <&i2c1>;
        __overlay__ {
            status = "okay";

            sgtl5000_codec: sgtl5000@0a {
                #sound-dai-cells = <0>;
                compatible = "fsl,sgtl5000";
                reg = <0x0a>;
                clocks = <&sgtl5000_mclk>;
                micbias-resistor-k-ohms = <2>;
                micbias-voltage-m-volts = <3000>;
                VDDA-supply = <&vdd_3v3_reg>;
                VDDIO-supply = <&vdd_3v3_reg>;
                VDDD-supply = <&reg_1v8>;
                status = "okay";
            };
        };
    };

    fragment@3 {
        target = <&i2s>;
        __overlay__ {
            status = "okay";
        };
    };

    fragment@4 {
        target = <&sound>;
        sound_overlay: __overlay__ {
           compatible = "simple-audio-card";
           simple-audio-card,format = "i2s";
           simple-audio-card,name = "sgtl5000";
           simple-audio-card,bitclock-master = <&dailink0_master>;
           simple-audio-card,frame-master = <&dailink0_master>;
           simple-audio-card,widgets =
               "Microphone", "Microphone Jack",
               "Speaker", "External Speaker";
           simple-audio-card,routing =
               "MIC_IN", "Mic Bias",
               "External Speaker", "LINE_OUT";
           status = "okay";
           simple-audio-card,cpu {
               sound-dai = <&i2s>;
           };
           dailink0_master: simple-audio-card,codec {
               sound-dai = <&sgtl5000_codec>;
           };
        };
    };
};

This is a much more complicated overlay, with a total of 5 fragments, which we will describe in detail:

  • fragment@0: this fragment adds a new Device Tree node that describes a clock that runs at a fixed frequency. Indeed, the sys_mclk clock of our codec is provided by an external oscillator running at 12.288 Mhz: this sgtl5000_mclk describes this external oscillator.
  • fragment@1: this fragment defines an external power supply used as the VDDD-supply of the sgtl5000 codec. Indeed, this codec is simply power by an always-on power supply at 1.8V. Don’t forget to enable both kernel options CONFIG_REGULATOR=y and CONFIG_REGULATOR_FIXED_VOLTAGE=y otherwise the driver will just silently fail to probe at boot.
  • fragment@2: this fragment describes the audio codec itself. The audio codec is described as a child of the I2C controller it is connected to: indeed the SGTL5000 uses I2C as its control interface.
  • fragment@3: this fragment simply enables the I2S interface. The CONFIG_SND_BCM2835_SOC_I2S=y kernel option must be enabled to have the corresponding driver.
  • fragment@4: this fragment describes the actual sound card. It uses the generic simple-audio-card driver, describes the two sides of the audio link: the CPU interface in the &i2s and the codec interface in the &sgtl_codec node, and describes the audio widgets and routes. See the simple-audio-card Device Tree binding

With this overlay in place, we need to enable it in the config.txt file, as well as the I2C1 overlay with a correct pin-muxing configuration:

dtparam=i2s=on
dtoverlay=sgtl5000
dtoverlay=i2c1,pins_2_3
# AUDIO_SD
gpio=40=op,dh

Once the system has booted, a new audio card is visible, and all the classic ALSA user-space utilities can be used: amixer to control the volume, aplay and arecord to play/record audio.

Conclusion

In this blog post, we have shown how Device Tree Overlays can easily be used to extend the description of the hardware, and enable the use of additional hardware devices on a Raspberry Pi system. Do not hesitate to contact us for support on Raspberry Pi platforms!

Live Embedded Event: a new online conference

Live Embedded EventIn these times of COVID19, pretty much all of the existing conferences have moved to an online format. For example, the Embedded Linux Conference Europe is going to take place next week, online, and Bootlin will significantly contribute to the event with no less than 7 talks on a wide range of topics.

But this trend for online conferences has also spurred the creation of new events. And specifically, we’re happy to announce the creation of a new conference oriented towards our favorite topic of embedded systems: Live Embedded Event. It will take place online on December 3 and will have a broader range of topics covered than ELC typically has, as Live Embedded Event is open to non-Linux embedded topics, hardware platform and interfaces discussions, and more.

The event is mainly organized by four French companies specialized in embedded systems, and we expect the contents to be a mix of talks in English and in French. Speaking of contents, the Call For Papers is open until November 10, so you can have the chance to share your knowledge about embedded topics, show your expertise, and meet new users and customers. And of course, you can also simply register as an attendee: the event is free !

Important links:

Linux 5.9 released: Bootlin contributions

Linux 5.9 was released last Sunday. See our usual resources for a good coverage of the highlights of this new release: KernelNewbies page, LWN.net article on the first part of the merge window, LWN.net article on the second part of the merge window.

On our side, we contributed a total of 69 commits to Linux 5.9, which unusually low and makes Bootlin the 31st contributing company by number of commits according to Linux Kernel Patch Statistic. The highlights of our contributions are:

  • Miquèl Raynal reworked part of the rawnand subsystem to allow drivers for non-ONFI compliant NANDs to select a more efficient data interface.
  • On the support of Atmel/Microchip platforms
    • Alexandre Belloni added proper support for the sama5d2 in the TCB (timer counter block) clocksource driver. While doing so, he upstreamd most of the remaining PREEMPT-RT patches for this driver.
    • Kamel Bouhara submitted a new driver for the TCB, this time in the counter subsystem, allowing to count external events, see our post on the topic
  • Antoine Ténart worked on PTP/IEEE 1588 support for the Microchip/Microsemi Ethernet PHYs. This required a change in the core to support a quad PHY properly. The series also included previous work from Quentin Schulz.
  • Paul Kocialkowski added support for the Rockchip PX30 SoC in the V4L2 M2M RGA driver.
  • Maxime Chretien, one of our trainee this year sent a fix for qconf.

Also, several Bootlin engineers are maintainers of various areas of the Linux kernel:

  • Miquèl Raynal, as the NAND maintainer and MTD co-maintainer, reviewed and merged 57 patches from other contributors
  • Alexandre Belloni, as the RTC maintainer and Microchip platform support co-maintainer, reviewed and merged 54 patches from other contributors
  • Grégory Clement, as the Marvell EBU platform support co-maintainer, reviewed and merged 13 patches from other contributors

Here is the complete list of our contributions:

Upcoming online training courses in November/December 2020

The online training courses that we started earlier this year continue to have a good success, and as our sessions scheduled for September/October are now completed or full, we are happy to announce our next set of dates for November/December:

All courses are taught by experienced Bootlin engineers, who are not just trainers: they in fact spend most of their time working on engineering projects for our customers, and share their experience through these training courses. The above courses will be taught by Michael Opdenacker, Alexandre Belloni, Maxime Chevallier, Paul Kocialkowski and Thomas Petazzoni.

Note that in addition to those public training courses, we can upon request organize dedicated online training courses, to match the needs, schedule and availability of our customers. Do not hesitate to contact us for any question.

Bootlin toolchains 2020.08 released

Bootlin toolchainsWe are happy to announce a new release of the freely available cross-compilation toolchains we provide at toolchains.bootlin.com, version 2020.08-1.

Here are the main changes compared to our previous 2020.02 release:

  • Bleeding edge toolchains are now using: gcc 10.2, binutils 2.34, gdb 9.2, kernel headers 5.4, glibc 2.31, musl 1.2.0, uclibc-ng 1.0.34
  • Stable toolchains are using: gcc 9.3, binutils 2.33, gdb 8.3, kernel headers 4.9, glibc 2.31, musl 1.2.0, uclibc-ng 1.0.34
  • Fortran support has been enabled in all tolchains
  • Several new CPU architecture variants are supported, each with a new toolchain
  • Boot testing in Qemu was added for PowerPC64 E5500, NIOSII and m68k MCF5208.

In addition, it is worth mentioning that all those Bootlin toolchains are now directly accessible from Buildroot: make menuconfig shows the Bootlin toolchains available for the current selected CPU architecture, and Buildroot is able to automatically download and use the toolchain. This feature will be available starting from Buildroot 2020.11:

Thanks again to the entire Buildroot community, and especially Romain Naour, for all the fixes and improvements related to toolchain support that make this project possible. In the next weeks, we hope to be able to deliver further updated bleeding-edge toolchains, with glibc 2.32 and binutils 2.35. Stay tuned!

If you face any issue, or need additional features in those toolchains, do not hesitate to report an issue in our issue tracker.

Supporting a misbehaving NAND ECC engine

Over the years, Bootlin has grown a significant expertise in U-Boot and Linux support for flash memory devices. Thanks to this expertise, we have recently been in charge of rewriting and upstreaming a driver for the Arasan NAND controller, which is used in a number of Xilinx Zynq SoCs. It turned out that supporting this NAND controller had some interesting challenges to handle its ECC engine peculiarities. In this blog post, we would like to give some background about ECC issues with NAND flash devices, and then dive into the specific issues that we encountered with the Arasan NAND controller, and how we solved them.

Ensuring data integrity

NAND flash memories are known to be intrinsically rather unstable: over time, external conditions or repetitive access to a NAND device may result in the data being corrupted. This is particularly true with newer chips, where the number of corruptions usually increases with density, requiring even stronger corrections. To mitigate this, Error Correcting Codes are typically used to detect and correct such corruptions, and since the calculations related to ECC detection and correction are quite intensive, NAND controllers often embed a dedicated engine, the ECC engine, to offload those operations from the CPU.

An ECC engine typically acts as a DMA master, moving, correcting data and calculating syndromes on the fly between the controller FIFO’s and the user buffer. The engine correction is characterized by two inputs: the size of the data chunks on which the correction applies and the strength of the correction. Old SLC (Single Level Cell) NAND chips typically require a strength of 1 symbol over 4096 (1 bit/512 bytes) while new ones may require much more: 8, 16 or even 24 symbols.

In the write path, the ECC engine reads a user buffer and computes a code for each chunk of data. NAND pages being longer than officially advertised, there is a persistent Out-Of-Band (OOB) area which may be used to store these codes. When reading data, the ECC engine gets fed by the data coming from the NAND bus, including the OOB area. Chunk by chunk, the engine will do some math and correct the data if needed, and then report the number of corrected symbols. If the number of error is higher than the chosen strength, the engine is not capable of any correction and returns an error.

The Arasan ECC engine

As explained in our introduction, as part of our work on upstreaming the Arasan NAND controller driver, we discovered that this NAND controller IP has a specific behavior in terms of how it reports ECC results: the hardware ECC engine never reports errors. It means the data may be corrected or uncorrectable: the engine behaves the same. From a software point of view, this is a critical flaw and fully relying on such hardware was not an option.

To overcome this limitation, we investigated different solutions, which we detail in the sections below.

Suppose there will never be any uncorrectable error

Let’s be honest, this hypothesis is highly unreliable. Besides that anyway, it would imply that we do not differentiate between written/erased pages and users would receive unclean buffers (with bitflips), which would not work with upper layers such as UBI/UBIFS which expect clean data.

Keep an history of bitflips of every page

This way, during a read, it would be possible to compare the evolution of the number of bitflips. If it suddenly drops significantly, the engine is lying and we are facing an error. Unfortunately it is not a reliable solution either because we should either trigger a write operation every time a read happens (slowing down a lot the I/Os and wearing out very quickly the storage device) or loose the tracking after every power cycle which would make this solution very fragile.

Add a CRC16

This CRC16 could lay in the OOB area and help to manually verify the data integrity after the engine’s correction by checking it against the checksum. This could be acceptable, even if not perfect in term of collisions. However, it would not work with existing data while there are many downstreams users of the vendor driver already.

Use a bitwise XOR between raw and corrected data

By doing a bitwise XOR between raw and corrected datra, and compare with the number of bitflips reported by the engine, we could detect if the engine is lying on the number of corrected bitflips. This solution has actually been implemented and tested. It involves extra I/Os as the page must be read twice: first with correction and then again without correction. Hence, the NAND bus throughput becomes a limiting factor. In addition, when there are too many bitflips, the engine still tries to correct data and creates bitflips by itself. The result is that, with just a XOR, we cannot discriminate a working correction from a failure. The following figure shows the issue.

Show the engine issue when it creates bitflips when trying to correct uncorrectable data

Rely on the hardware only in the write path

Using the hardware engine in the write path is fine (and possibly the quickest solution). Instead of trying to workaround the flaws of the read path, we can do the math by software to derive the syndrome in the read path and compare it with the one in the OOB section. If it does not match, it means we are facing an uncorrectable error. This is finally the solution that we have chosen. Of course, if we want to compare software and hardware calculated ECC bytes, we must find a way to reproduce the hardware calculations, and this is what we are going to explore in the next sections.

Reversing a hardware BCH ECC engine

There is already a BCH library in the Linux kernel on which we could rely on to compute BCH codes. What needed to be identified though, were the BCH initial parameters. In particular:

  • The BCH primary polynomial, from which is derived the generator polynomial. The latter is then used for the computation of BCH codes.
  • The range of data on which the derivation would apply.

There are several thousands possible primary polynomials with a form like x^3 + x^2 + 1. In order to represent these polynomials more easily by software, we use integers or binary arrays. In both cases, each bit represents the coefficient for the order of magnitude corresponding to its position. The above example could be represented by b1101 or 0xD.

For a given desired BCH code (ie. the ECC chunk size and hence its corresponding Gallois Field order), there is a limited range of possible primary polynomials which can be used. Given eccsize being the amount of data to protect, the Gallois Field order is the smallest integer m so that: 2^m > eccsize. Knowing m, one can check these tables to see examples of polynomials which could match (non exhaustive). The Arasan ECC engine supporting two possible ECC chunk sizes of 512 and 1024 bytes, we had to look at the tables for m = 13 and m = 14.

Given the required strength t, the number of needed parity bits p is: p = t x m.

The total amount of manipulated data (ECC chunk, parity bits, eventual padding) n, also called BCH codeword in papers, is: n = 2^m - 1.

Given the size of the codeword n and the number of parity bits p, it is then possible to derive the maximum message length k with: k = n - p.

The theory of BCH also shows that if (n, k) is a valid BCH code, then (n - x, k - x) will also be valid. In our situation this is very interesting. Indeed, we want to protect eccsize number of symbols, but we currently cover k within n. In other words we could use the translation factor x being: x = k - eccsize. If the ECC engine was also protecting some part of the OOB area, x should have been extended a little bit to match the extra range.

With all this theory in mind, we used GNU Octave to brute force the BCH polynomials used by the Arasan ECC engine with the following logic:

  • Write a NAND page with a eccsize-long ECC step full of zeros, and another one full of ones: this is our known set of inputs.
  • Extract each BCH code of p bits produced by the hardware: this is our known set of outputs.

For each possible primary polynomial with the Gallois Field order m, we derive a generator polynomial, use it to encode both input buffers thanks to a regular BCH derivation, and compare the output syndromes with the expected output buffers.

Because the GNU Octave program was not tricky to write, we first tried to match with the output of Linux software BCH engine. Linux using by default the primary polynomial which is the first in GNU Octave’s list for the desired field order, it was quite easy to verify the algorithm worked.

As unfortunate as it sounds, running this test with the hardware data did not gave any match. Looking more in depth, we realized that visually, there was something like a matching pattern between the output of the Arasan engine and the output of Linux software BCH engine. In fact, both syndromes where identical, the bits being swapped at byte level by the hardware. This observation was made possible because the input buffers have the same values no matter the bit ordering. By extension, we also figured that swapping the bits in the input buffer was also necessary.

The primary polynomial for an eccsize of 512 bytes being already found, we ran again the program with eccsize being 1024 bytes:

eccsize = 1024
eccstrength = 24
m = 14
n = 16383
p = 336
k = 16047
x = 7855
Trying primary polynomial #1: 0x402b
Trying primary polynomial #2: 0x4039
Trying primary polynomial #3: 0x4053
Trying primary polynomial #4: 0x405f
Trying primary polynomial #5: 0x407b
[...]
Trying primary polynomial #44: 0x43c9
Trying primary polynomial #45: 0x43eb
Trying primary polynomial #46: 0x43ed
Trying primary polynomial #47: 0x440b
Trying primary polynomial #48: 0x4443
Primary polynomial found! 0x4443

Final solution

With the two possible primary polynomials in hand, we could finish the support for this ECC engine.

At first, we tried a “mixed-mode” solution: read and correct the data with the hardware engine and then re-read the data in raw mode. Calculate the syndrome over the raw data, derive the number of roots of the syndrome which represents the number of bitflips and compare with the hardware engine’s output. As finding the syndrome’s roots location (ie. the bitflips offsets) is very time consuming for the machine it was decided not to do it in order to gain some time. This approach worked, but doing the I/Os twice was slowing down very much the read speed, much more than expected.

The final approach has been to actually get rid of any hardware computation in the read path, delegating all the work to Linux BCH logic, which indeed worked noticeably faster.

The overall work is now in the upstream Linux kernel:

If you’re interested about more details on ECC for flash devices, and their support in Linux, we will be giving a talk precisely on this topic at the upcoming Embedded Linux Conference!

Bootlin at the Embedded Linux Conference Europe 2020

Embedded Linux Conference Europe 2020The schedule for the next Embedded Linux Conference Europe has been recently published, and Bootlin will once again be strongly present at this (virtual) event by giving a number of presentations. The registration for ELC-E is open, and due to the virtual nature of the event, the registration cost is only $50, which makes is accessible to pretty much everybody.

  • From the Camera Sensor to the User, the Journey of a Video Frame. In this talk, Bootlin engineer Maxime Chevallier will share his experience working in the Video4Linux subsystem, implementing support for the Rockchip camera interface controller, and a PAL/NTSC decoder used as the input source. Talk on Monday October 26 at 18:30 GMT.
  • Yocto Project and OpenEmbedded: A Collection of Best Practices. In this talk, Bootlin engineer, and Yocto Project expert and trainer Alexandre Belloni will share his experience of using Yocto Project and OpenEmbedded through a collection of best practices. There are indeed numerous ways of using OpenEmbedded and the Yocto Project, but some ways and solutions are better than others! Talk on Tuesday October 27 at 13:00 GMT.
  • Building Embedded Debian and Ubuntu Systems with ELBE. In this talk, Köry Maincent will share his experience using the ELBE build system, which can be used to automate the process of creating embedded Linux systems based on Debian or Ubuntu. Köry has contributed to ELBE the support for building Ubuntu systems, and has used ELBE on two different projects. This is an interesting alternative to the traditional cross-compilation approach taken by Yocto Project, OpenEmbedded or Buildroot. Talk on Tuesday October 27 at 15:15 GMT.
  • Using Visual Studio Code for Embedded Development. In this talk, Michael Opdenacker will share his experiments of using VS Code for embedded Linux development, which extensions are available to help navigate in the code, build and debug code, from kernel-space and user-space. Talk on Tuesday October 27 at 17:15 GMT.
  • Precision Time Protocol (PTP) and Packet Timestamping in Linux. Antoine Ténart has implemented PTP support in both an Ethernet switch driver, and an Ethernet PHY driver, both in the upstream Linux kernel. He will share his experience about PTP, its support in Linux, and its offloading at the MAC and PHY level. Talk on Tuesday October 27 at 19:30 GMT.
  • Supporting Hardware-Accelerated Video Encoding with Mainline. After working on the HW-accelerated video decoding on Allwinner platforms as part of our crowdfunded effort, Paul Kocialkowski recently worked on HW-accelerated video encoding on Rockchip platforms. In this talk, he will share the issues encountered, and what needs to be resolved to create a useful kernel to userspace interface to properly support stateless video encoders. Talk on Wednesday October 28 at 16:15 GMT.
  • Understand ECC Support for NAND Flash Devices in Linux. Miquèl Raynal, the Linux kernel NAND subsystem maintainer, has recently worked on improving support for various strategies to handle ECC for NAND flash devices. He will share some background information on ECC, why they are needed, how and where ECC are typically handled, and how the Linux kernel deals with the different possibilities. Talk on Wednesday October 28 at 18:30 GMT.

In addition to contributing talks, Bootlin CTO Thomas Petazzoni is also a member of the ELC-E program committee: he reviewed, ranked all talk submitted for the conference and participated with the rest of the committee to the selection of the talks that are now scheduled for the event.

Even though we once again won’t have the chance to meet our fellow members of the embedded Linux community in person, we look forward to attending a set of great talks, and have interesting discussions during the Q&A and through the instant messaging platform that will be available around the conference.