Bootlin contributions to Linux 5.13

After finally publishing about our Linux 5.12 contributions and even though Linux 5.14 was just released yesterday, it’s hopefully still time to talk about our contributions to Linux 5.13. Check out the LWN articles about the merge window to get the bigger picture about this release: part 1 and part 2.

In terms of Bootlin contributions, this was a much more quiet release than Linux 5.12, with just 28 contributions. The main highlights are:

  • The usual round of RTC subsystem updates from its maintainer Alexandre Belloni
  • A large amount of improvements in the MTD subsystem by its co-maintainer Miquèl Raynal, continuing his effort to improve the ECC handling in the MTD subsystem. See Miquèl’s talk at ELCE 2020 for more details on this effort: slides and video.
  • A small fix for an annoying regression in the musb USB gadget controller driver.

Even though we contributed just 28 commits to this release, as maintainers, some of us also reviewed and merged code from other contributors: Miquèl Raynal as the MTD co-maintainer merged 63 patches, Alexandre Belloni merged 22 patches, and Grégory Clement 6 patches.

Here are the details of our contributions to Linux 5.13:

Mainline Linux support for the ARM Primecell PL35X NAND controller

It has been more than 7 years since the first draft of a Linux kernel driver for the ARM Primecell PL35X NAND controller was posted on a public mailing list. Maybe because of the lack of time, each new version was delayed so much that it actually needed another iteration just to catch up with the latest internal API changes in the MTD subsystem (quite a number of them happened in the last 2-3 years). The NAND controller itself is part of an ARM Primecell Static Memory bus Controller (SMC) which increased the overall complexity. Finally, the way the commands and data are shared with the memory controller is very specific to the SMC. All these technical points probably played against Xilinx engineers, and Bootlin was contracted in 2021 to finalize the work of getting the ARM Primecell PL35X NAND controller driver in the upstream Linux kernel.

Static Memory Controller principles

SMC diagram from the TRM

The SMC can interface with two different memory types: NAND or SRAM/NOR. As it features two memory slots, this means that it can drive two memories, but they must be of the same type. When handling NAND devices, a hardware ECC engine is available to perform on-the-fly correction.

As only a single type of memory device can be plugged in at a time (either two SRAM/NORs or two NANDs), we don’t need to share a lot of controls with the SRAM/NOR controllers. So in the end the memory bus driver is almost an empty envelope that relies on the child controller driver to do the job.

Interactions with a memory device

On the CPU side, the controller has two main interfaces: APB and AXI.

The APB interface works like any regular interface: the CPU sees registers that it can access with diverse read and write operations, which will effectively read and write the content of the 32-bit registers located at the desired addresses. This is how the driver configures the device type, the timings, the possible ECC configuration and so forth. All the initial SMC configuration is done through the APB interface.

The AXI interface does not quite work like this. Instead of featuring a set of registers at a fixed address in which the content of the command, address and data cycles would be written in order to be forwarded to the memory device, the AXI interface needs to reserve a notable range in the addressable space. In particular, the offset targetted by the AXI write depend on the type of action that must be performed and the content of the action:

  • When requesting the controller to send command and address cycles to the memory device, the datasheet refers to it as the “command phase”.
  • When doing I/Os, eg. actually reading from/writing to the memory device, the datasheet calls this the “data phase”.

Both the command and data phase use regular AXI read/writes, but the offsets and values are different than usual.

Command phase

When the driver wants to send command cycles, it must perform one or two register writes. The address of the write operation in the AXI address space must target a specific offset. This offset indicates a number of information:

  • A specific bit is set to tell the SMC that it must enter a command phase.
  • Part of the offset are made of the shifted values of the different command opcodes for the memory device.
  • Part of the offset encodes the number of address cycles to perform on the NAND bus.

The payload of the AXI write contains the value of the address cycles that should be forwarded to the memory device. If there are more than 4 address cycles (which is quite common today), then a second AXI write containing the remaining address cycles as payload must happen at the same offset as before.

/*
 * Define the offset in the AXI address space where to write with:
 * - the bit indicating the command phase
 * - the number of address cycles
 * - the command opcode
 */
cmd_addr = PL35X_SMC_CMD_PHASE |
           PL35X_SMC_CMD_PHASE_NADDRS(naddr_cycles) |
           PL35X_SMC_CMD_PHASE_CMD0(NAND_CMD_XXXX);

/* Define the payload with the address bytes */
for (i = 0, row = 0; row < nrows; i++, row++) {
        [...]
        if (row < 4)
                addr1 |= PL35X_SMC_CMD_PHASE_ADDR(row, addr);
        else
                addr2 |= PL35X_SMC_CMD_PHASE_ADDR(row - 4, addr);
}

/* Send the command and address cycles */
writel(addr1, nfc->io_regs + cmd_addr);
if (naddr_cycles > 4)
        writel(addr2, nfc->io_regs + cmd_addr);

Data phase

The data phase is a bit easier to understand: several AXI reads or writes will be performed at a specific offset. The payload matches our expectations: it is actually the data that we want to read from or write to the device. However, the offset in the AXI address space is again a bit counter-intuitive:

  • It contains a specific bit such as the command phase to inform the controller that the data phase must be entered.
  • It also contains shifted values of different flags regarding the ECC configuration. The thing is, this offset will change at the end of the I/O operation because the last chunk of data must always be handled differently because of the ECC calculations that must be manually started. We end up reading or writing physically contiguous data by accessing two completely different offsets.
/* I/O transfers: simple case */
for (i = 0; i < buf_end; i++) {
        data_phase_addr = PL35X_SMC_DATA_PHASE;
        if (i + 1 == buf_end)
                data_phase_addr += PL35X_SMC_DATA_PHASE_ECC_LAST;

        writel(buf32[i], nfc->io_regs + data_phase_addr);
}

But what happens if a command cycle must be sent at the end of a data transfer (typical case of a PAGE_WRITE)? While it would certainly be more logical to perform an additional command phase AXI write, it was certainly more optimized to merge data and command phase on the last access. And here is how it looks like:

/* I/O transfers: less straightforward situation */
for (i = 0; i < buf_end; i++) {
        data_phase_addr = PL35X_SMC_DATA_PHASE;
        if (i + 1 == buf_end)
                data_phase_addr +=
                    PL35X_SMC_DATA_PHASE_ECC_LAST |
                    PL35X_SMC_CMD_PHASE_CMD1(NAND_CMD_PAGEPROG) |
                    PL35X_SMC_CMD_PHASE_CMD1_VALID);

        writel(buf32[i], nfc->io_regs + data_phase_addr);
}

Of course, nothing highly unreadable, but at the very least, these accesses are quite uncommon.

A memory bus driver and a NAND controller driver

As explained earlier, this SMC controller can support different types of memories, and this has called for a Device Tree representation where the SMC controller is one node, and the memories connected to it are represented as sub-node. So, the Device Tree representation of the SMC controller, used with its NAND controller looks like this:

    smcc: memory-controller@e000e000 {
      compatible = "arm,pl353-smc-r2p1", "arm,primecell";
      reg = <0xe000e000 0x0001000>;
      clock-names = "memclk", "apb_pclk";
      clocks = <&clkc 11>, <&clkc 44>;
      ranges = <0x0 0x0 0xe1000000 0x1000000 /* Nand CS region */
                0x1 0x0 0xe2000000 0x2000000 /* SRAM/NOR CS0 region */
                0x2 0x0 0xe4000000 0x2000000>; /* SRAM/NOR CS1 region */
      #address-cells = <2>;
      #size-cells = <1>;

      nfc0: nand-controller@0,0 {
        compatible = "arm,pl353-nand-r2p1";
        reg = <0 0 0x1000000>;
        #address-cells = <1>;
        #size-cells = <0>;
      };
    };

So, we first have a node for the SMC controller itself, memory-controller@e000e000, which will allow probing the memory bus driver located at drivers/memory/pl353-smc.c. This driver is very simple: it enables the clocks necessary for the SMC to work, and then it probes the first child device that matches either the cfi-flash or arm,pl353-nand-r2p1 compatible strings. In the latter case (which is illustrated in our example), the NAND controller driver at drivers/mtd/nand/raw/pl35x-nand-controller.c will be probed, and where the two memory areas (accessed through APB and AXI) will be mapped, and accessed to program the NAND controller.

Now in the mainline Linux kernel

Starting from the latest version posted by Xilinx, Miquèl Raynal, Bootlin’s NAND/MTD expert, performed a massive cleanup of the memory bus driver and the NAND controller driver, rewrote entirely the binding file (in YAML schema!) and three versions later, with the support of Xilinx engineers and the acknowledgements of Rob Herring and Krzysztof Kozlowski, managed to finally close the story. The driver is now part of Linux 5.14-rc, and will therefore be in the final Linux 5.14 release in a few weeks!

GPIO Aggregator, a virtual gpio chip

GPIOs are obviously widely used in embedded systems, and many of them are typically driven directly by Linux kernel drivers for interrupt lines, reset lines, or other control lines used to connect with various peripherals. However, a number of GPIOs are sometimes directly driven by user-space applications. Historically, the Linux kernel has provided a sysfs interface, in /sys/class/gpio to allow such direct control. But in recent years, this sysfs interface has been superseded by a new user-space interface based on /dev/gpiochip* character devices.

This new interface has numerous advantages over the previous /sys/class/gpio interface. However, one drawback is that it creates one device file per GPIO chip, which means that access rights are defined per GPIO chip, and not per GPIOs.

For this reason, in Linux 5.8, Geert Uytterhoeven has contributed the GPIO aggregator mechanism. It allows to group a number of GPIOs into a virtual GPIO chip, visible as an additional /dev/gpiochip*. Its documentation can be found in Documentation/admin-guide/gpio/gpio-aggregator.rst.

The list of GPIOs part of this new virtual GPIO chip is defined in the Device Tree. One other interesting thing is that, as any GPIO controler, the lines can be named, and then queried by user-space applications based on their name, using the libgpiod library.

Configuration and Device Tree description

To have GPIO Aggregator support in your kernel, simply configure

CONFIG_GPIO_AGGREGATOR=y

Add a gpio-aggregator node in your Device Tree source. For instance, the following DTS snippet declares an aggregator with several GPIO lines:

gpio-aggregator {
    pinctrl-names = "default";
    pinctrl-0 = <&gpio_pins>;
    compatible = "gpio-aggregator";

    gpios = <&gpio3 4 GPIO_ACTIVE_HIGH>,
            <&gpio2 4 GPIO_ACTIVE_HIGH>,
            <&gpio1 28 GPIO_ACTIVE_HIGH>,
            <&gpio1 29 GPIO_ACTIVE_HIGH>,
            <&gpio2 0 GPIO_ACTIVE_HIGH>,
            <&gpio2 1 GPIO_ACTIVE_HIGH>,
            <&gpio3 8 GPIO_ACTIVE_HIGH>;

    gpio-line-names = "line_a", "line_b", "line_c", "line_d",
            "line_e", "line_f", "line_g";
};

In this example, line 4 of gpio controller gpio3 is used and is named “line_a”, line 4 of gpio controller gpio2 is used and is named “line_b”, and so on up to line 8 of gpio controler gpio3.

Usage from user-space

From userspace we can see the GPIO chip and its aggregated lines:

# gpioinfo
...
gpiochip6 - 7 lines:
    line 0: "line_a" unused input active-high
    line 1: "line_b" unused input active-high
    line 2: "line_c" unused input active-high
    line 3: "line_d" unused input active-high
    line 4: "line_e" unused input active-high
    line 5: "line_f" unused input active-high
    line 6: "line_g" unused input active-high
...

We can search a gpio chip and a line number by the line name:

# gpiofind 'line_b'
gpiochip6 1

We can access a GPIO line by its name

# gpioget $(gpiofind 'line_b')
1
#
# gpioset $(gpiofind 'line_e')=1
# gpioset $(gpiofind 'line_e')=0

We can change the GPIO chip device file ownership to allow user or group to access the attached lines:

# ls -al /dev/gpiochip*
crw------- 1 root root 254, 0 Jan 1 00:00 /dev/gpiochip0
crw------- 1 root root 254, 1 Jan 1 00:00 /dev/gpiochip1
crw------- 1 root root 254, 2 Jan 1 00:00 /dev/gpiochip2
crw------- 1 root root 254, 3 Jan 1 00:00 /dev/gpiochip3
crw------- 1 root root 254, 4 Jan 1 00:00 /dev/gpiochip4
crw------- 1 root root 254, 5 Jan 1 00:00 /dev/gpiochip5
crw------- 1 root root 254, 6 Jan 1 00:00 /dev/gpiochip6
#
# chown root:users /dev/gpiochip6
# chmod 660 /dev/gpiochip6
# ls -al /dev/gpiochip*
crw------- 1 root root 254, 0 Jan 1 00:00 /dev/gpiochip0
crw------- 1 root root 254, 1 Jan 1 00:00 /dev/gpiochip1
crw------- 1 root root 254, 2 Jan 1 00:00 /dev/gpiochip2
crw------- 1 root root 254, 3 Jan 1 00:00 /dev/gpiochip3
crw------- 1 root root 254, 4 Jan 1 00:00 /dev/gpiochip4
crw------- 1 root root 254, 5 Jan 1 00:00 /dev/gpiochip5
crw-rw---- 1 root users 254, 6 Jan 1 00:00 /dev/gpiochip6
#

The GPIO chip created by the aggregator can be retrieved from sysfs:

# ls -1 /sys/bus/platform/devices/gpio-aggregator/
driver
driver_override
gpio
gpiochip6
modalias
of_node
power
subsystem
uevent
# 
# cat /sys/bus/platform/devices/gpio-aggregator/gpiochip6/dev
254:6
#

Conclusion

A GPIO Aggregator can be used to group a subset of GPIO lines, name them, access them by their names and manage access control to the virtual gpio chip created by the aggregator. On an embedded system, this can simplify the management and usage of individual GPIO lines.

Bootlin contributions to Linux 5.12

Yes, Linux 5.13 was released yesterday, but we never published the blog post detailing our contributions to Linux 5.12, so let’s do this now! First of all the usual links to the excellent LWN.net articles on the 5.12 merge window: part 1 and part 2.

LWN.net also published an article with Linux 5.12 development statistics, and two Bootlin engineers made their way to the statistics: Alexandre Belloni in the list of top contributors by number of changesets, with 69 commits, and Paul Kocialkowski in the list of top contributors by number of changed lines, with over 6000 lines changed.

Here are the highlights of our contributions:

  • Addition of a new driver for the Silvaco I3C master controller. This was contributed by Miquèl Raynal, who became the maintainer for this driver. Bootlin has pioneered support for I3C in Linux, by introducing the complete drivers/i3c subsystem a few years ago, together with the first controller driver, for a Cadence IP, see our blog post from 2018.
  • Addition of two new camera sensor drivers, one for the Omnivision OV5648 and another for the Omnivision OV8865. These were contributed by Paul Kocialkowski.
  • Implementation of mqprio support in the Marvell Ethernet controller driver mvneta, see this commit. As explained in the tc-mqprio man page, the MQPRIO qdisc is a simple queuing discipline that allows mapping traffic flows to hardware queue ranges using priorities and a configurable priority to traffic class mapping. This was contributed by Maxime Chevallier
  • Improvements in the IIO driver for the ms58xx family of sensors, contributed by Alexandre Belloni.
  • The final removal of the atmel_tclib code, which has been replaced by proper drivers for the TCB timers on Atmel/Microchip ARM platforms over the past few releases, also by Alexandre Belloni.
  • As usual, a large amount of fixes and improvements in the RTC subsystem, by its maintainer Alexandre Belloni.

Here is the detailed list of our contributions to this release:

LTP: Linux Test Project, Bootlin contributions

Introduction

The Linux Test Project is a project that develops and maintains a large test suite that helps validating the reliability, robustness and stability of the Linux kernel and related features. LTP has been mainly developed by companies such as IBM, Cisco, Fujitsu, SUSE, RedHat, with a focus on desktop distributions.

On the embedded side, both the openembedded-core Yocto layer and Buildroot have packages that allow to use LTP on embedded targets. However, for a recent project, we practically tried to run the full LTP test suite on an i.MX8 based platform running a Linux system built with Yocto. It turned out that LTP was apparently not very often tested on Busybox-based embedded systems, and we faced a number of issues. In addition to reporting various bugs/issues to the upstream LTP project, we also contributed a number of fixes and improvements:

Our contributions received a very warm welcome in the LTP community, which turned out to be very open and responsive. We hope that these contributions will encourage others to use LTP, and hopefully to make sure it continues to work on embedded platforms.

Quick start guide

At the time of this writing, LTP has more than 3800 tests written by the community, including about 1000 network-related tests. The tests are grouped together in categories described by files in the runtest/ folder. Based on this, two scenarios of tests are defined: default and network which are described by two files in the scenario_groups/ folder. These two scenarios simply list the categories of tests that need to be executed.

Here are the contents of the default and network:

$ cat scenario_groups/default 
syscalls
fs
fs_perms_simple
fsx
dio
io
mm
ipc
sched
math
nptl
pty
containers
fs_bind
controllers
filecaps
cap_bounds
fcntl-locktests
connectors
power_management_tests
hugetlb
commands
hyperthreading
can
cpuhotplug
net.ipv6_lib
input
cve
crypto
kernel_misc
uevent
$ cat scenario_groups/network 
can
net.features
net.ipv6
net.ipv6_lib
net.tcp_cmds
net.multicast
net.rpc
net.nfs
net.rpc_tests
net.tirpc_tests
net.sctp
net_stress.appl
net_stress.broken_ip
net_stress.interface
net_stress.ipsec_dccp
net_stress.ipsec_icmp
net_stress.ipsec_sctp
net_stress.ipsec_tcp
net_stress.ipsec_udp
net_stress.multicast
net_stress.route

Once you have LTP built and installed on your board thanks to the appropriate OpenEmbedded or Buildroot package, you can run these two scenarios of test with the following commands (-n specify the network one):

$ cd /opt/ltp
$ ./runltp
$ ./runltp -n

Then take a look at the content of the result and the output directories.

For more information on building or running LTP please read this readme.