Bringing NV-DDR support to parallel NAND flashes in Linux

We have recently contributed support for NV-DDR interfaces to parallel NAND flashes in the Linux kernel, which brings performance improvements for a number of NAND flash devices. In this article, we will detail what are the ONFI specifications, the historical SDR interface, then the introduction of faster interfaces in the ONFI specification, and finally our work to support such interfaces in the Linux kernel.

ONFI specifications

Even though specifications came after the introduction of NAND devices on the market, the Open NAND Flash Interface (ONFI) specification is nowadays a de-facto specification which many NAND chip support (even non-ONFI ones). For instance, in the Linux kernel, we assume that any NAND flash device will by default, after a reset command, at least support the slowest set of ONFI timings. Other specifications exist, like the Joint Electron Device Engineering Council (JEDEC), but as it is a bit less common in the parallel NAND flashes world, we will focus on the ONFI details in this blog post.

The early days of the SDR interface

At the time of the first ONFI specification back in 2006, there was only a single interface detailed: the asynchronous data interface. Also known as Single Data Rate or SDR interface in modern language, it defines the timings sequence that should be respected in order for any NAND controller to be able to deal with almost any kind of NAND device. As an asynchronous interface, in this interface, the data bus has no clock signal. Instead, it features a specific set of signals which are asserted by the controller to signal read data latch and write data latch: Read Enable (RE#) and Write Enable (WE#).

The data interface can work in 6 different timing modes, from 0 to 5. 0 is the slowest mode and the default one at boot time with a theoretical data rate of about 10MiB/s (assuming an 8-bit bus). Mode 4 and 5 are the fastest, they leverage the ability of Extended Data Output (EDO) to latch data on both RE#/WE# edges and may reach a theoretical data rate of 50MiB/s.

The introduction of faster interfaces

Shortly after, at the beginning of 2008, the ONFI consortium released the second version of the ONFI specification and included a new interface: the source synchronous data interface. This interface is backward compatible with the asynchronous interface and allows the host to switch from one interface to the other if this is needed. In the particular case of the source synchronous interface, a clock (CLK) signal is replacing the legacy WE# signal and indicates when the commands and address should be latched. The direction of the transfers is handled by the Write/Read signal (W/R#) in place of RE# signal. Finally, a data strobe (DQS) signal is being introduced and indicates when the data should be latched. As both edges of the DQS signal advertise for a data latch, the source synchronous interface is also called Double Data Rate (DDR) interface even though this naming was only introduced in the version 3.0 of the specification, in 2011.

The exact terms that are used in more recent specifications are NV-DDR (Non-Volatile DDR), NV-DDR2 and NV-DDR3 which are backward compatible improvements of the NV-DDR interface. For instance, the first NV-DDR specification has a range of theoretical rates from 40MiB/s to 200MiB/s.

ONFI datasheet on data interfaces

Support in the Linux kernel

While the addition of the MTD/NAND subsystem in the Linux kernel predates the Git era and is now over 20 years old, Linux users have always been limited to use the asynchronous interface (SDR modes). At Bootlin, we recently started an effort to bring support for the NV-DDR interface to the Linux kernel MTD/NAND subsystem, and this involved the following changes:

  • Introducing an API to propose timings to the host controller driver, so that it might either accept or refuse them (only SDR mode 0 cannot be refused) and be aware of all timings that this choice involves so that the host controller registers will be configured properly.
  • Adding the possibility for NAND chip drivers to tweak the timings if the parameter page is not present or inaccurate.
  • Adding the core logic to ask the NAND chip to change its data interface through the use of GET_FEATURE and SET_FEATURE calls, as well as verifying that this operation worked correctly and handling the fallback in case of error.

We recently reached a final step in this effort as the last missing parts will be part of the next Linux kernel release (v5.14). This final series aiming at bringing NV-DDR support to Linux carries the following changes:

  • Adding the necessary bits to parse the parameter page of the NAND device in order to know which NV-DDR modes the chips support.
  • Providing the reference implementation of all NV-DDR timing modes and various helpers to manage them.
  • Adding the necessary infrastructure and helpers to the host controller drivers in order to allow them to distinguish between SDR and NV-DDR, as well as advertise which mode they are willing to support based on the controller’s constraints.
  • Updating the existing logic to take into account the existence of NV-DDR timings and select them when appropriate. This part is a bit trickier as the core must gracefully fallback to SDR modes under certain conditions.

Overall, thanks to the major cleanups which happened in the NAND subsystem in the last three years, it was pretty straightforward to add support for these new timings.

Future work

It is worth mentioning that accelerating the overall throughput on the data bus without a deeper rework of the MTD core than just enabling faster timings is very limiting: data reads must respect a tR delay before starting and writes are considered effective only after a tPROG delay. Both are significantly high in practice: respectively about 25-45us and 200-600us, compared to the time needed to store/fetch the data through the I/O bus: a few dozens of micro-seconds.

To fully leverage the power of NV-DDR timings the NAND and MTD cores should be partially rewritten to bring parallel multi-die support and cached operations. Such features would allow to optimize the use of the I/O bus in order to mitigate the performances impact of tR and tPROG during massive I/O operations. This is precisely one of the tricks used by SSD drives to exhibit very fast I/Os while using multiple NAND chips behind. There is therefore interesting additional work to do in the Linux kernel MTD subsystem to fully benefit from NV-DDR interfaces.

CFP for the Embedded Linux Conference 2021 ends June 13th

Initially planned to take place in Dublin, Ireland, the unique edition this year of the Embedded Linux Conference will take place in Seattle, US and virtually from September 27 to September 30, 2021. See also the conference website. Bootlin CEO Thomas Petazzoni is again a member of the program committee for this edition of ELC.

Embedded Linux Conference 2021

This kind of event is only possible thanks to the talks proposed by its participants! As detailed on the Call For Papers, the last date to submit your proposals is June 13, 2021. There is really a wide range of suggested topics, and ELC is an excellent place to talk about advancements in the Linux kernel for embedded platforms, in user-space libraries and stacks relevant to embedded, about practical experiences in using Linux in embedded devices, about real-time, boot time, power management, build systems, open hardware, and more.

We look forward to seeing your proposals for ELC!

Bootlin “Buildroot system development” course updated to Buildroot 2021.02

Bootlin has been offering for several years a Buildroot system development course, which allows engineers interested in learning and understanding the Buildroot embedded Linux build system to get up to speed very quickly.

In preparation for our public Buildroot system development course next week, we updated our training materials, both slides and labs to Buildroot 2021.02, which is the latest stable Buildroot release as of today, and is also a Long Term Support release.

Buildroot slide

In addition to updating to a newer Buildroot version, we also use newer U-Boot and Linux versions for the practical labs on BeagleBone Black Wireless. The slides were also updated to document some new features that appeared between 2020.02 and 2021.02. If you’re interested, check out the materials on the training page.

We have one seat left for this training course next week, which will be taught by long-time Buildroot contributor and developer Thomas Petazzoni. Register now and take the last seat!

Live Embedded Event schedule published, 5 talks from Bootlin

The schedule for the next edition of Live Embedded Event has been published! This 100% online and free conference will take place on June 3rd, 2021. Thanks to the proposals received, the event will feature 4 tracks during the entire day, covering a wide range of topics: hardware for embedded systems, embedded Linux, RTOS, IoT, FPGA, RISC-V, and more.

Live Embedded Event #2 agenda
Live Embedded Event #2 agenda

Bootlin is once again part of the organization team for this event, and in addition 5 talks proposed by Bootlin have been selected into the schedule. See below the details of our talks.

Understanding U-Boot Falcon Mode and adding support for new boards, Michael Opdenacker

The Falcon Mode is a U-Boot feature that allows to directly load the operating system kernel from the first stage of U-Boot (a.k.a. “SPL”), skipping the second stage of U-Boot. Doing this can save up to 1 second in the boot process, and this way, you can keep a full featured U-Boot that you can still fall back to for maintenance or development needs. However, using Falcon Mode is not always easy, as it requires extra code that most boards supported by U-Boot don’t have yet. At Bootlin, we had to add such support to U-Boot for several boards. This presentation will explain how Falcon Mode booting actually works in U-Boot and the implementation and usage choices made by U-Boot developers. It will show you how to add such Falcon Mode support to U-Boot for your own board.

Banner LEE for Michael Opdenacker's talk

Talk given by Michael Opdenacker, at 10:00 AM CEST on June 3rd, 2021.

Link to the talk (registration required).

Introduction to RAUC, Kamel Bouhara

In embedded systems, deploying firmware updates in the field has now become an obvious requirement, to ensure that security vulnerabilities are addressed, that bugs are fixed, and new functionalities can be delivered to the users. Among a range of different open-source solutions, RAUC provides an interesting firmware update mechanism for embedded system. In this talk, we will introduce the main features of RAUC, its integration in build systems such as Buildroot or the Yocto Project, as well as its integration with the U-Boot and Barebox bootloaders. Finally we will explore some common update scenarios that are fully supported by RAUC features.

Banner LEE for Kamel Bouhara's talk

Talk given by Kamel Bouhara, at 3:30 PM CEST on June 3rd, 2021.

Link to the talk (registration required).

Security vulnerability tracking tools in Buildroot, Thomas Petazzoni

Buildroot is a popular and easy to use embedded Linux build system. With the increasing concern around security vulnerabilities affecting embedded systems, and the need to keep them updated, Buildroot has been extended with new tooling for security vulnerability tracking. This tooling allows to monitor the CVEs that affect the packages present in Buildroot. In this talk, we will introduce the principle of CVEs and CPEs, present the tools now available in Buildroot to help keep track of the security vulnerabilities, show how they can be used for a project and identify the current limitations of this tooling.

Banner for LEE's talk from Thomas Petazzoni

Talk given by Thomas Petazzoni, at 1:30 PM CEST on June 3rd, 2021.

Link to the talk (registration required).

Secure boot in embedded Linux systems, Thomas Perrot

Secure boot is a integrity mechanism, based on signature verification, that allows to detect software corruption or malicious code, during the boot process. Implementing secure boot is not always obvious, as it requires multiple stages of verification, at the bootloader, Linux kernel and root filesystem level, as well as integration into the build system, CI infrastructure, firmware upgrade mechanism, and more. Based on a recent experience to bring secure boot on an NXP i.MX8 platform, Thomas will present how to implement the chain of trust from the SoC ROM code to the root filesystem, as well as other considerations related to the implementation of secure boot. While the presentation will use the i.MX8 as an example, most of the discussion will apply to other platforms as well.

Banner for LEE's talk from Thomas Perrot

Talk given by Thomas Perrot, at 3:30 PM CEST on June 3rd, 2021.

Link to the talk (registration required).

Device Tree overlays and U-boot extension board management, Köry Maincent

In this talk, we will start by introducing the mechanism of Device Tree Overlays, which are a way of extending the Device Tree itself to describe additional hardware. We will show how Device Tree Overlays are written, compiled, and applied to a base Device Tree, and what is the status of Device Tree Overlays support in U-Boot and Linux. We will take the example of the BeagleBoard.org project, showing how Device Tree overlays are used to make CAPE extension boards compatible with different boards. Finally, we will describe our proposal, already submitted to the community, to add an extension board management facility to U-Boot, which automatically detects, loads and applies the appropriate Device Tree Overlays depending on the extension boards that are detected.

Banner for LEE's talk from Köry Maincent

Talk given by Köry Maincent, at 1:30 PM CEST on June 3rd, 2021.

Link to the talk (registration required).

Using Buildroot to flash and boot the beta version of BeagleV Starlight

Bootlin recently received a beta prototype of the BeagleV Starlight featuring a RISC-V 64 bit SoC capable of running Linux, designed by StarFive This early version is not available to the general public, but several of us at Bootlin volunteered to join the beta developer program to assist with upstream software development. BeagleBoard.org has a public BeagleV forum that everyone can join for future updates on the project.

Two days after my colleague Thomas Petazzoni received his board, he managed to submit a patch for the mainline version of Buildroot to add support for this new board. Actually, compiling an image with Buildroot and preparing an SD card is easier than downloading and flashing the initial Fedora image offered for this beta board.

If you are just interested in testing the software on your board, you may directly get our binaries from our Build results paragraph.

The following instructions are derived from the board/beaglev/readme.txt file in Thomas’ proposed patch.

How to build

First, clone Buildroot’s git repository if you haven’t done it yet:

$ git clone git://git.buildroot.net/buildroot

Then add a remote branch corresponding to Thomas Petazzoni’s own tree, as his changes haven’t made their way into the mainline yet, and checkout a local copy of his beaglev branch:

$ git remote add tpetazzoni https://github.com/tpetazzoni/buildroot.git
$ git fetch tpetazzoni
$ git checkout -b tpetazzoni-beaglev tpetazzoni/beaglev

Now you can build the binaries for the board:

$ make beaglev_defconfig
$ make

Build results

After building, output/images should contain the following files:

  • Image
  • fw_payload.bin
  • fw_payload.bin.out
  • fw_payload.elf
  • rootfs.ext2
  • rootfs.ext4
  • sdcard.img
  • u-boot.bin

The two important files are:

  • fw_payload.bin.out, which is the bootloader image, containing both OpenSBI (the Open Supervisor Binary Interface, allowing to switch from Machine mode to Supervisor mode) and U-Boot.
  • sdcard.img, the SD card image, which contains the root filesystem, kernel image and Device Tree.

Tested versions of these generated files are available on our website.

Flashing the SD card image

You just need to insert your micro SD card into a card reader (assuming the /dev/sdX device file is used), and type the below command:

$ sudo dd if=output/images/sdcard.img of=/dev/sdX

Preparing the board

To prevent the experimental board from overheating, connect the BeagleV fan to the 5V supply (pin 2 or 4 of the GPIO connector) and GND (pin 6 of the GPIO connector).

To access a serial console, connect a TTL UART cable to pins 6 (GND), 8 (TX) and 10 (RX):
Beagle V - How to connect the serial port

Insert your SD card and power-up the board using a USB-C cable.

Flashing the bootloader

The bootloader pre-flashed on the BeagleV has a non-working fdt_addr_r environment variable value, so it won’t work as-is. Reflashing the existing bootloader with the bootloader image produced by Buildroot is therefore necessary.

When the board starts up, a pre-loader shows a count down of 2 seconds. Interrupt it by pressing any key. You should then reach a menu like
this:

bootloader version:210209-4547a8d
ddr 0x00000000, 1M test
ddr 0x00100000, 2M test
DDR clk 2133M,Version: 210302-5aea32f
0
***************************************************
*************** FLASH PROGRAMMING *****************
***************************************************

0:update uboot
1:quit
select the function:

Press 0 and Enter. You will now see C characters being displayed. Ask your serial port communication program to send the fw_payload.bin.out file using the Xmodem protocol (with the sx command). For example, here’s how to do it with picocom

picocom should be started as:

$ picocom -b 115200 -s "sx -vv" /dev/ttyUSB0

When you see the C characters on the serial line, press [Ctrl][a] [Ctrl][s]. Picocom will then ask for a file name, and you should type fw_payload.bin.out.

After a few minutes, reflashing should be complete. Then, restart the board. It will automatically start the system from the SD card, and reach the login prompt:

Welcome to Buildroot
buildroot login: root
# uname -a
Linux buildroot 5.10.6 #2 SMP Sun May 2 17:23:56 CEST 2021 riscv64 GNU/Linux

Useful resources

Here are useful resources for people who already have the Beagle V board:

We will keep updating this page according to progress in the upstream projects:

  • Support for the board in mainline Buildroot
  • Later, support for the board in mainline U-Boot and Linux