Linux 4.15 released, Bootlin contributions

Penguin from Mylène Josserand
Drawing from Mylène Josserand,
based on a picture from Samuel Blanc under CC-BY-SA

After a month of February busier than usual, with the renaming of our company from Free Electrons to Bootlin, our participation to FOSDEM and the welcoming of Maxime Chevallier, the latest addition to our engineering team, our article on the latest release of the Linux kernel arrives a bit late, more than a month after Linux 4.15 has been released by Linus Torvalds.

As usual, LWN.net did an interesting coverage of this release cycle merge window, highlighting the most important changes: The first half of the 4.15 merge window and The rest of the 4.15 merge window. Due to the now well-known Spectre and Meltdown vulnerabilities and the resulting effort to try to mitigate them, 4.15 required a -rc9, which happened the last time back in 2011 with the 3.1, Torvalds said.

According to Linux Kernel Patch statistics, Bootlin (now Bootlin) contributed 150 patches to this release, making it the 16th contributing company by number of commits.

The main highlights of our contributions are:

  • In the RTC subsystem, Alexandre Belloni made a number of improvements to various drivers, mainly making them use the nvmem subsystem where appropriate, and use the recently introduced rtc_register_device() API.
  • In the MTD subsystem, both Boris Brezillon and Miquèl Raynal made a number of contributions, mainly fixes.
  • For Marvell platforms
    • Antoine Ténart contributed a few fixes to the inside-secure crypto accelerator driver, used on Marvell Armada 3700 and Armada 7K/8K
    • Antoine Ténart also contributed fixes and improvements to the mvpp2 network driver, used for the Ethernet controller on the Marvell Armada 7K/8K. His improvements include preparation work to support Receive Side Scaling (RSS).
    • Antoine Ténart enabled more networking ports and features in some Armada 7K/8K boards, especially SFP ports on Armada 7040 DB and Armada 7040 DB.
    • Boris Brezillon contributed a few fixes to the Marvell CESA crypto accelerator driver, used on the older Orion, Kirkwood, Armada 370/XP/38x processors. He migrated the driver to use the skcipher interface of the Linux kernel crypto framework.
    • Grégory Clement enabled NAND support on Armada 7K, and contributed a number of fixes around MMC support for some Marvell boards.
    • Thomas Petazzoni contributed a few minor Device Tree enhancements for Marvell platforms: fixing MPP muxing on an older Kirkwood platform, enabling more PCIe ports on Armada 8040 DB, etc.
    • Miquèl Raynal contributed support for more advanced statistics in the mvpp2 network driver.
    • Miquèl Raynal added support for the extended UART for the Marvell Armada 3720 processor, both in the UART driver and in the Device Tree.
  • For the RaspberryPi platform, Boris Brezillon contributed a few fixes to the vc4 display driver, and added support for the new DRM_IOCTL_VC4_GEM_MADVISE ioctl, which can be used to ask the userspace applications to purge inactive buffers when allocations start to fail in the kernel.
  • For Allwinner platforms
    • Mylène Josserand contributed a fix for the Allwinner A83 clock driver, fixing I2C bus clocks.
    • Quentin Schulz contributed a few fixes to the sun4i-gpadc-iio.c driver, which is used for the ADCs on several Allwinner processors.
    • Maxime Ripard made a number of fixes to the sun8i-codec driver, fixing clock issues, left/right channels inversion, etc.
    • Maxime Ripard made a number of improvements to the sun4i DRM display driver.
    • Maxime Ripard improved the support for the A83 processor (described the UART1 controller, the MMC1 controller, added support for display clocks) and added the Device Tree for a new A83 device.
    • Maxime Ripard also did a number of cleanups and misc improvements in a significant number of Device Tree files for Allwinner platforms.
  • Thomas Petazzoni made a few fixes to the sh_eth network driver, used on several Renesas SuperH platform, as part of a recent project Bootlin did on SuperH 4.

Bootlin engineers are not only contributors, but also maintainers of various subsystems in the Linux kernel, which means they are involved in the process of reviewing, discussing and merging patches contributed to those subsystems:

  • Maxime Ripard, as the Allwinner platform co-maintainer, merged 108 patches from other contributors
  • Boris Brezillon, as the MTD/NAND maintainer, merged 34 patches from other contributors
  • Alexandre Belloni, as the RTC maintainer and Atmel platform co-maintainer, merged 50 patches from other contributors
  • Grégory Clement, as the Marvell EBU co-maintainer, merged 24 patches from other contributors

Here is the commit by commit detail of our contributons to 4.15:

Bootlin contributes a new interface to the Linux NAND subsystem

MTD stack

Over the last months, Bootlin engineers Boris Brezillon and Miquèl Raynal have been working on rewriting the NAND controller driver used on a large number of Marvell SoCs. This NAND controller driver had grown very complicated, and Miquèl’s adventure in this rework led him to contribute a new interface to the NAND framework, in order to simplify implementing NAND controller drivers for complex NAND controllers. In this blog post, Miquèl summarizes the original issue, and how it is solved by the ->exec_op() interface he has contributed.

Introduction

The NAND framework is the layer between the generic MTD layer and the NAND controller drivers. Its purpose is to handle MTD requests and transform them into understandable NAND operations the controller will have to send to the NAND chip.

For general information about NANDs, the reader is invited to read the ONFI specification (Open NAND Flash Interface) which defines the most common NAND operations.

Interacting with a NAND chip

Raw NANDs (so-called “parallel NANDs”) are slave devices waiting for instructions from the controller. An operation is a sequence of instructions usually referred as “command” (CMD), “addresses” (ADDR), and “data” cycles (DATA_IN/DATA_OUT) and sometimes wait periods (WAITRDY). Some everyday operations any NAND enthusiast should know by heart are, for instance:

NAND operation example

How it was handled in the Linux kernel

Today, a majority of NAND controlller drivers implement the ->cmd_ctrl() hook. It aimed to be a very small function, designed to just send command and address cycles independently, usually embedding some very controller-specific logic. This hook was supposed to be called by a function of higher level from the NAND core, ->cmdfunc(). In addition to calling ->cmd_ctrl() to send command and address cycles, the core would also call ->read|write_byte|word|buf() hooks to actually move data from the NAND controller and the memory (the DATA parts in the diagram above).

This approach worked very well with simple NAND controllers, which are just able to send command and address cycles one at a time to the NAND chip, without any extra intelligence. However, NAND controllers have become more and more complex and now can handle higher-level operations, usually to provide higher performance. For example, a NAND controller may provide an operation that would do all of the command and address cycles of a read-page operation in one-go. Some controllers even support only those higher-level operations, and are not able to simply do the basic operation of sending one command cycle or one data cycle. To handle such controllers, their drivers were overloading the ->cmdfunc() hook directly, circumventing the generic NAND core implementation of ->cmdfunc(). This is a first drawback: it is no longer possible to easily add logic to the NAND core to support new NAND operations, because some drivers overload the ->cmdfunc() logic. Worse, ->cmdfunc() doesn’t provide some information such as the length of the data transfer, which some controllers actually need in order to run the desired operation. NAND controller drivers started to have complicated state machines just to work around the NAND framework limitations.

NAND stack before exec_op

Some driver-specific implementations of this hook started diverging from the original one, giving maintainers a lot of pain to maintain the whole subsystem, specifically when they needed to introduce additional vendor-specific operations support. These implementations were not only diverse but also incomplete, sometimes buggy and most importantly, developers had to guess the data that would probably be moved by the core after that, which is clearly a symptom that the framework was not fitting the user needs anymore.

The ->exec_op() era

The NAND subsystem maintainers decided to switch to a new approach, based on a new hook called ->exec_op(), implemented by NAND controller drivers and called by the generic NAND core. The logic behind that name is to provide to every controller a generic interface that can easily be extended and exposes the overall NAND operation to be performed. This way, the driver can optimize depending on the controller capabilities without the need of a complex state machine as ->cmdfunc() was.

All major NAND generic raw operations like reset, reading the NAND ID, selecting a set of timings, reading/writing data and so on found their place into small internal functions named nand_[operation]_op().

From the NAND controller driver point of view, an array of instructions is received for each operation. The controller then needs to parse these instructions, decides if it can handle the overall operation, splits the operation if needed, and executes what is requested.

Using the ->exec_op() interface is as simple as declaring a list with the controller capabilities, each entry of this array having a callback function knowing the overall operation that will actually handle all the logic. The NAND core was enhanced with a proper parser that one may use in his driver to handle the callback selection logic.

NAND stack with exec_op

For a more complete overview, one can check the slides and the video of Miquèl’s presentation at FOSDEM about NAND flash memories and the introduction of ->exec_op() in the Linux kernel.

Current status

The ->exec_op() interface in the NAND core has been accepted and merged upstream, and will be part of Linux 4.16. The first driver converted to this new interface was obviously the NAND controller driver used on Marvell platforms, pxa3xx_nand. It has been rewritten as marvell_nand, and will also be part of Linux 4.16. Even though the new driver is longer (by lines of code) than the previous one, it supports additional features (such as raw read and write operations), allows the NAND core to pass custom commands to the NAND chip, and has a logic that is a lot less complicated.

Miquèl has also worked on converting the fsmc_nand driver to ->exec_op(), but this work hasn’t been merged yet. In the community, Stefan Agner has taken on the task to convert the vf610_nfc driver to this new approach.

Bootlin is proud to have contributed such enhancements to the Linux kernel, and hopes to see other developers contribute to this subsystem in the near future, by migrating their favorite NAND controller driver to ->exec_op()!

Linux 4.14 released, Bootlin contributions

Penguin from Mylène Josserand
Drawing from Mylène Josserand,
based on a picture from Samuel Blanc under CC-BY-SA
Linux 4.14, which is going to become the next Long Term Supported version, has been released a week ago by Linus Torvalds. As usual, LWN.net did an interesting coverage of this release cycle merge window, highlighting the most important changes: The first half of the 4.14 merge window and The rest of the 4.14 merge window.

According to Linux Kernel Patch statistics, Bootlin contributed 111 patches to this release, making it the 24th contributing company by number of commits: a somewhat lower than usual contribution level from our side. At least, Bootlin cannot be blamed for trying to push more code into 4.14 because of its Long Term Support nature! 🙂

The main highlights of our contributions are:

  • On the RTC subsystem, Alexandre Belloni made as usual a number of fixes and improvements to various drivers, especially the ds1307 driver.
  • On the NAND subsystem, Boris Brezillon did a number of small improvements in various areas.
  • On the support for Marvell platforms
    • Antoine Ténart improved the ppv2 network driver used by the Marvell Armada 7K/8K SoCs: support for 10G speed and TSO support are the main highlights. In order to support 10G speed, Antoine added a driver in drivers/phy/ to configure the common PHYs in the Armada 7K/8K SoCs.
    • Thomas Petazzoni also improved the ppv2 network driver by adding support for TX interrupts and per-CPU RX interrupts.
    • Grégory Clement contributed some patches to enable NAND support on Armada 7K/8K, as well as a number of fixes in different areas (GPIO fix, clock handling fixes, etc.)
    • Miquèl Raynal contributed a fix for the Armada 3700 SPI controller driver.
  • On the support for Allwinner platforms
    • Maxime Ripard contributed the support for a new board, the BananaPI M2-Magic. Maxime also contributed a few fixes to the Allwinner DRM driver, and a few other misc fixes (clock, MMC, RTC, etc.).
    • Quentin Schulz contributed the support for the power button functionality of the AXP221 (PMIC used in several Allwinner platforms)
  • On the support for Atmel platforms, Quentin Schulz improved the clock drivers for this platform to properly support the Audio PLL, which allowed to fix the Atmel audio drivers. He also fixed suspend/resume support in the Atmel MMC driver to support the deep sleep mode of the SAMA5D2 processor.

In addition to making direct contributions, Bootlin is also involved in the Linux kernel development by having a number of its engineers act as Linux kernel maintainers. As part of this effort, Bootlin engineers have reviewed, merged and sent pull requests for a large number of contributions from other developers:

  • Boris Brezillon, as the NAND subsystem maintainer and MTD subsystem co-maintainer, merged 68 patches from other developers.
  • Alexandre Belloni, as the RTC subsystem maintainer and Atmel ARM platform co-maintainer, merged 32 patches from other developers.
  • Grégory Clement, as the Marvell ARM platform co-maintainer, merged 29 patches from other developers.
  • Maxime Ripard, as the Allwinner ARM platform co-maintainer, merged 18 patches from other developers.

This flow of patches from kernel maintainers to other kernel maintainers is also nicely described for the 4.14 release by the Patch flow into the mainline for 4.14 LWN.net article.

The detailed list of our contributions:

Linux 4.13 released, Bootlin contributions

Linux 4.13 was released last Sunday by Linus Torvalds, and the major new features of this release were described in details by LWN in a set of articles: part 1 and part 2.

This release gathers 13006 non-merge commits, amongst which 239 were made by Bootlin engineers. According to the LWN article on 4.13 statistics, this makes Bootlin the 13th contributing company by number of commits, the 10th by lines changed.

The most important contributions from Bootlin for this release have been:

  • In the RTC subsystem
    • Alexandre Belloni introduced a new method for registering RTC devices, with one step for the allocation, and one step for the registration itself, which allows to solve race conditions in a number of drivers.
    • Alexandre Belloni added support for exposing the non-volatile memory found in some RTC devices through the Linux kernel nvmem framework, making them usable from userspace. A few drivers were changed to use this new mechanism.
  • In the MTD/NAND subsystem
    • Boris Brezillon did a large number of fixes and minor improvements in the NAND subsystem, both in the core and in a few drivers.
    • Thomas Petazzoni contributed the support for on-die ECC, specifically with Micron NANDs. This allows to use the ECC calculation capabilities of the NAND chip itself, as opposed to using software ECC (calculated by the CPU) or ECC done by the NAND controller.
    • Thomas Petazzoni contributed a few improvements to the FSMC NAND driver, used on ST Spear platforms. The main improvement is to support the ->setup_data_interface() callback, which allows to configure optimal timings in the NAND controller.
  • Support for Allwinner ARM platforms
    • Alexandre Belloni improved the sun4i PWM driver to use the so-called atomic API and support hardware read out.
    • Antoine Ténart improved the sun4i-ss cryptographic engine driver to support the Allwinner A13 processor, in addition to the already supported A10.
    • Maxime Ripard contributed HDMI support for the Allwinner A10 processor (in the DRM subsystem) and a number of related changes to the Allwinner clock support.
    • Quentin Schulz improved the support for battery charging through the AXP20x PMIC, used on Allwinner platforms.
  • Support for Atmel ARM platforms
    • Alexandre Belloni added suspend/resume support for the Atmel SAMA5D2 clock driver. This is part of a larger effort to implement the backup mode for the SAMA5D2 processor.
    • Alexandre Belloni added suspend/resume support in the tcb_clksrc driver, used as for clocksource and clockevents on Atmel SAMA5D2.
    • Alexandre Belloni cleaned up a number of drivers, removing support for non-DT probing, which is possible now that the AVR32 architecture has been dropped. Indeed, the AVR32 processors used to share the same drivers as the Atmel ARM processors.
    • Alexandre Belloni added the core support for the backup mode on Atmel SAMA5D2, a suspend/resume state with significant power savings.
    • Boris Brezillon switched Atmel platforms to use the new binding for the EBI and NAND controllers.
    • Boris Brezillon added support for timing configuration in the Atmel NAND driver.
    • Quentin Schulz added suspend/resume support to the Bosch m_can driver, used on Atmel platforms.
  • Support for Marvell ARM platforms
    • Antoine Ténart contributed a completely new driver (3200+ lines of code) for the Inside Secure EIP197 cryptographic engine, used in the Marvell Armada 7K and 8K processors. He also subsequently contributed a number of fixes and improvements for this driver.
    • Antoine Ténart improved the existing mvmdio driver, used to communicate with Ethernet PHYs over MDIO on Marvell platforms to support the XSMI variant found on Marvell Armada 7K/8K, used to communicate with 10G capable PHYs.
    • Antoine Ténart contributed minimal support for 10G Ethernet in the mvpp2 driver, used on Marvell Armada 7K/8K. For now, the driver still relies on low-level initialization done by the bootloader, but additional changes in 4.14 and 4.15 will remove this limitation.
    • Grégory Clement added a new pinctrl driver to configure the pin-muxing on the Marvell Armada 37xx processors.
    • Grégory Clement did a large number of changes to the clock drivers used on the Marvell Armada 7K/8K processors to prepare the addition of pinctrl support.
    • Grégory Clement added support for Marvell Armada 7K/8K to the existing mvebu-gpio driver.
    • Thomas Petazzoni added support for the ICU, a specialized interrupt controller used on the Marvell Armada 7K/8K, for all devices located in the CP110 part of the processor.
    • Thomas Petazzoni removed a work-around to properly resume per-CPU interrupts on the older Marvell Armada 370/XP platforms.
  • Support for RaspberryPi platforms
    • Boris Brezillon added runtime PM support to the HDMI encoder driver used on RaspberryPi platforms, and contributed a few other fixes to the VC4 DRM driver.

It is worth mentioning that Miquèl Raynal, recently hired by Bootlin, sees his first kernel patch merged: nand: fix wrong default oob layout for small pages using soft ecc.

Bootlin engineers are not only contributors, but also maintainers of various subsystems in the Linux kernel, which means they are involved in the process of reviewing, discussing and merging patches contributed to those subsystems:

  • Maxime Ripard, as the Allwinner platform co-maintainer, merged 113 patches from other contributors
  • Boris Brezillon, as the MTD/NAND maintainer, merged 62 patches from other contributors
  • Alexandre Belloni, as the RTC maintainer and Atmel platform co-maintainer, merged 57 patches from other contributors
  • Grégory Clement, as the Marvell EBU co-maintainer, merged 47 patches from other contributors

Here is the commit by commit detail of our contributors to 4.13:

Bootlin proposes an I3C subsystem for the Linux kernel

MIPI I3C fact sheet, from the MIPI I3C white paper
MIPI I3C fact sheet, from the MIPI I3C white paper
At the end of 2016, the MIPI consortium has finalized the first version of its I3C specification, a new communication bus that aims at replacing older busses like I2C or SPI. According to the specification, I3C gets closer to SPI data rate while requiring less pins and adding interesting mechanisms like in-band interrupts, hotplug capability or automatic discovery of devices connected on the bus. In addition, I3C provides backward compatibility with I2C: I3C and legacy I2C devices can be connected on a common bus controlled by an I3C master.

For more details about I3C, we suggest reading the MIPI I3C Whitepaper, as unfortunately MIPI has not publicly released the specifications for this protocol.

For the last few months, Bootlin engineer Boris Brezillon has been working with Cadence to develop a Linux kernel subsystem to support this new bus, as well as Cadence’s I3C master controller IP. We have now posted the first version of our patch series to the Linux kernel mailing list for review, and we already received a large number of very useful comments from the kernel community.

Bootlin is proud to be pioneering the support for this new bus in the Linux kernel, and hopes to see other developers contribute to this subsystem in the near future!

Linux 4.12, Bootlin contributions

Linus Torvalds has released the 4.12 Linux kernel a week ago, in what is the second biggest kernel release ever by number of commits. As usual, LWN had a very nice coverage of the major new features and improvements: first part, second part and third part.

LWN has also published statistics about the Linux 4.12 development cycles, showing:

  • Bootlin as the #14 contributing company by number of commits, with 221 commits, between Broadcom (230 commits) and NXP (212 commits)
  • Bootlin as the #14 contributing company number of changed lines, with 16636 lines changed, just two lines less than Mellanox
  • Bootlin engineer and MTD NAND maintainer Boris Brezillon as the #17 most active contributor by number of lines changed.

Our most important contributions to this kernel release have been:

  • On Atmel AT91 and SAMA5 platforms:
    • Alexandre Belloni has continued to upstream the support for the SAMA5D2 backup mode, which is a very deep suspend to RAM state, offering very nice power savings. Alexandre touched the core code in arch/arm/mach-at91 as well as pinctrl and irqchip drivers
    • Boris Brezillon has converted the Atmel PWM driver to the atomic API of the PWM subsystem, implemented suspend/resume and did a number of fixes in the Atmel display controller driver, and also removed the no longer used AT91 Parallel ATA driver.
    • Quentin Schulz improved the suspend/resume hooks in the atmel-spi driver to support the SAMA5D2 backup mode.
  • On Allwinner platforms:
    • Mylène Josserand has made a number of improvements to the sun8i-codec audio driver that she contributed a few releases ago.
    • Maxime Ripard added devfreq support to dynamically change the frequency of the GPU on the Allwinner A33 SoC.
    • Quentin Schulz added battery charging and ADC support to the X-Powers AXP20x and AXP22x PMICs, found on Allwinner platforms.
    • Quentin Schulz added a new IIO driver to support the ADCs found on numerous Allwinner SoCs.
    • Quentin Schulz added support for the Allwinner A33 built-in thermal sensor, and used it to implement thermal throttling on this platform.
  • On Marvell platforms:
    • Antoine Ténart contributed Device Tree changes to describe the cryptographic engines found in the Marvell Armada 7K and 8K SoCs. For now only the Device Tree description has been merged, the driver itself will arrive in Linux 4.13.
    • Grégory Clement has contributed a pinctrl and GPIO driver for the Marvell Armada 3720 SoC (Cortex-A53 based)
    • Grégory Clement has improved the Device Tree description of the Marvell Armada 3720 and Marvell Armada 7K/8K SoCs and corresponding evaluation boards: SDHCI and RTC are now enabled on Armada 7K/8K, USB2, USB3 and RTC are now enabled on Armada 3720.
    • Thomas Petazzoni made a significant number of changes to the mvpp2 network driver, finally adding support for the PPv2.2 version of this Ethernet controller. This allowed to enable network support on the Marvell Armada 7K/8K SoCs.
    • Thomas Petazzoni contributed a number of fixes to the mv_xor_v2 dmaengine driver, used for the XOR engines on the Marvell Armada 7K/8K SoCs.
    • Thomas Petazzoni cleaned-up the MSI support in the Marvell pci-mvebu and pcie-aardvark PCI host controller drivers, which allowed to remove a no-longer used MSI kernel API.
  • On the ST SPEAr600 platform:
    • Thomas Petazzoni added support for the ADC available on this platform, by adding its Device Tree description and fixing a clock driver bug
    • Thomas did a number of small improvements to the Device Tree description of the SoC and its evaluation board
    • Thomas cleaned up the fsmc_nand driver, which is used for the NAND controller driver on this platform, removing lots of unused code
  • In the MTD NAND subsystem:
    • Boris Brezillon implemented a mechanism to allow vendor-specific initialization and detection steps to be added, on a per-NAND chip basis. As part of this effort, he has split into multiple files the vendor-specific initialization sequences for Macronix, AMD/Spansion, Micron, Toshiba, Hynix and Samsung NANDs. This work will allow in the future to more easily exploit the vendor-specific features of different NAND chips.
  • Other contributions:
    • Maxime Ripard added a display panel driver for the ST7789V LCD controller

In addition, several Bootlin engineers are also maintainers of various kernel subsystems. During this release cycle, they reviewed and merged a number of patches from kernel contributors:

  • Maxime Ripard, as the Allwinner co-maintainer, merged 94 patches
  • Boris Brezillon, as the NAND maintainer and MTD co-maintainer, merged 64 patches
  • Alexandre Belloni, as the RTC maintainer and Atmel co-maintainer, merged 38 patches
  • Grégory Clement, as the Marvell EBU co-maintainer, merged 32 patches

The details of all our contributions for this release:

Beyond boot testing: custom tests with LAVA

Since April 2016, we have our own automated testing infrastructure to validate the Linux kernel on a large number of hardware platforms. We use this infrastructure to contribute to the KernelCI project, which tests every day the Linux kernel. However, the tests being done by KernelCI are really basic: it’s mostly booting a basic Linux system and checking that it reaches a shell prompt.

However, LAVA, the software component at the core of this testing infrastructure, can do a lot more than just basic tests.

The need for custom tests

With some of our engineers being Linux maintainers and given all the platforms we need to maintain for our customers, being able to automatically test specific features beyond a simple boot test was a very interesting goal.

In addition, manually testing a kernel change on a large number of hardware platforms can be really tedious. Being able to quickly send test jobs that will use an image you built on your machine can be a great advantage when you have some new code in development that affects more than one board.

We identified two main use cases for custom tests:

  • Automatic tests to detect regression, as does KernelCI, but with more advanced tests, including platform specific tests.
  • Manual tests executed by engineers to validate that the changes they are developing do not break existing features, on all platforms.

Overall architecture

Several tools are needed to run custom tests:

  • The LAVA instance, which controls the hardware platforms to be tested. See our previous blog posts on our testing hardware infrastructrure and software architecture
  • An appropriate root filesystem, that contains the various userspace programs needed to execute the tests (benchmarking tools, validation tools, etc.)
  • A test suite, which contains various scripts executing the tests
  • A custom test tool that glues together the different components

The custom test tool knows all the hardware platforms available and which tests and kernel configurations apply to which hardware platforms. It identifies the appropriate kernel image, Device Tree, root filesystem image and test suite and submits a job to LAVA for execution. LAVA will download the necessary artifacts and run the job on the appropriate device.

Building custom rootfs

When it comes to test specific drivers, dedicated testing, validation or benchmarking tools are sometimes needed. For example, for storage device testing, bonnie++ can be used, while iperf is nice for networking testing. As the default root filesystem used by KernelCI is really minimalist, we need to build our owns, one for each architecture we want to test.

Buildroot is a simple yet efficient tool to generate root filesystems, it is also used by KernelCI to build their minimalist root filesystems. We chose to use it and made custom configuration files to match our needs.

We ended up with custom rootfs built for ARMv4, ARMv5, ARMv7, and ARMv8, that embed for now Bonnie++, iperf, ping (not the Busybox implementation) and other tiny tools that aren’t included in the default Buildroot configuration.

Our Buildroot fork that includes our custom configurations is available as the buildroot-ci Github project (branch ci).

The custom test tool

The custom test tool is the tool that binds the different elements of the overall architecture together.

One of the main features of the tool is to send jobs. Jobs are text files used by LAVA to know what to do with which device. As they are described in LAVA as YAML files (in the version 2 of the API), it is easy to use templates to generate them based on a single model. Some information is quite static such as the device tree name for a given board or the rootfs version to use, but other details change for every job such as the kernel to use or which test to run.

We made a tool able to get the latest kernel images from KernelCI to quickly send jobs without having a to compile a custom kernel image. If the need is to test a custom image that is built locally, the tool is also able to send files to the LAVA server through SSH, to provide a custom kernel image.

The entry point of the tool is ctt.py, which allows to create new jobs, providing a lot of options to define the various aspects of the job (kernel, Device Tree, root filesystem, test, etc.).

This tool is written in Python, and lives in the custom_tests_tool Github project.

The test suite

The test suite is a set of shell scripts that perform tests returning 0 or 1 depending on the result. This test suite is included inside the root filesystem by LAVA as part of a preparation step for each job.

We currently have a small set of tests:

  • boot test, which simply returns 0. Such a test will be successful as soon as the boot succeeds.
  • mmc test, to test MMC storage devices
  • sata test, to test SATA storage devices
  • crypto test, to do some minimal testing of cryptographic engines
  • usb test, to test USB functionality using mass storage devices
  • simple network test, that just validates network connectivity using ping

All those tests only require the target hardware platform itself. However, for more elaborate network tests, we needed to get two devices to interact with each other: the target hardware platform and a reference PC platform. For this, we use the LAVA MultiNode API. It allows to have a test that spans multiple devices, which we use to perform multiple iperf sessions to benchmark the bandwidth. This test has therefore one part running on the target device (network-board) and one part running on the reference PC platform (network-laptop).

Our current test suite is available as the test_suite Github project. It is obviously limited to just a few tests for now, we hope to extend the tests in the near future.

First use case: daily tests

As previously stated, it’s important for us to know about regressions introduced in the upstream kernel. Therefore, we have set up a simple daily cron job that:

  • Sends custom jobs to all boards to validate the latest mainline Linux kernel and latest linux-nextli>
  • Aggregates results from the past 24 hours and sends emails to subscribed addresses
  • Updates a dashboard that displays results in a very simple page
A nice dashboard showing the tests of the Beaglebone Black and the Nitrogen6x.

Second use case: manual tests

The custom test tool ctt.py has a simple command line interface. It’s easy for someone to set it up and send custom jobs. For example:

ctt.py -b beaglebone-black -m network

will start the network test on the BeagleBone Black, using the latest mainline Linux kernel built by KernelCI. On the other hand:

ctt.py -b armada-7040-db armada-8040-db -t mmc --kernel arch/arm64/boot/Image --dtb-folder arch/arm64/boot/dts/

will run the mmc test on the Marvell Armada 7040 and Armada 8040 development boards, using the locally built kernel image and Device Tree.

The result of the job is sent over e-mail when the test has completed.

Conclusion

Thanks to this custom test tool, we now have an infrastructure that leverages our existing lab and LAVA instance to execute more advanced tests. Our goal is now to increase the coverage, by adding more tests, and run them on more devices. Of course, we welcome feedback and contributions!

Linux 4.11, Bootlin contributions

Linus Torvalds has released this Sunday Linux 4.11. For an overview of the new features provided by this new release, one can read the coverage from LWN: part 1, part 2 and part 3. The KernelNewbies site also has a detailed summary of the new features.

With 137 patches contributed, Bootlin is the 18th contributing company according to the Kernel Patch Statistics. Bootlin engineer Maxime Ripard appears in the list of top contributors by changed lines in the LWN statistics.

Our most important contributions to this release have been:

  • Support for Atmel platforms
    • Alexandre Belloni improved suspend/resume support for the Atmel watchdog driver, I2C controller driver and UART controller driver. This is part of a larger effort to upstream support for the backup mode of the Atmel SAMA5D2 SoC.
    • Alexandre Belloni also improved the at91-poweroff driver to properly shutdown LPDDR memories.
    • Boris Brezillon contributed a fix for the Atmel HLCDC display controller driver, as well as fixes for the atmel-ebi driver.
  • Support for Allwinner platforms
    • Boris Brezillon contributed a number of improvements to the sunxi-nand driver.
    • Mylène Josserand contributed a new driver for the digital audio codec on the Allwinner sun8i SoC, as well a the corresponding Device Tree changes and related fixes. Thanks to this driver, Mylène enabled audio support on the R16 Parrot and A33 Sinlinx boards.
    • Maxime Ripard contributed numerous improvements to the sunxi-mmc MMC controller driver, to support higher data rates, especially for the Allwinner A64.
    • Maxime Ripard contributed official Device Tree bindings for the ARM Mali GPU, which allows the GPU to be described in the Device Tree of the upstream kernel, even if the ARM kernel driver for the Mali will never be merged upstream.
    • Maxime Ripard contributed a number of fixes for the rtc-sun6i driver.
    • Maxime Ripard enabled display support on the A33 Sinlinx board, by contributing a panel driver and the necessary Device Tree changes.
    • Maxime Ripard continued his clean-up effort, by converting the GR8 and sun5i clock drivers to the sunxi-ng clock infrastructure, and converting the sun5i pinctrl driver to the new model.
    • Quentin Schulz added a power supply driver for the AXP20X and AXP22X PMICs used on numerous Allwinner platforms, as well as numerous Device Tree changes to enable it on the R16 Parrot and A33 Sinlinx boards.
  • Support for Marvell platforms
    • Grégory Clement added support for the RTC found in the Marvell Armada 7K and 8K SoCs.
    • Grégory Clement added support for the Marvell 88E6141 and 88E6341 Ethernet switches, which are used in the Armada 3700 based EspressoBin development board.
    • Romain Perier enabled the I2C controller, SPI controller and Ethernet switch on the EspressoBin, by contributing Device Tree changes.
    • Thomas Petazzoni contributed a number of fixes to the OMAP hwrng driver, which turns out to also be used on the Marvell 7K/8K platforms for their HW random number generator.
    • Thomas Petazzoni contributed a number of patches for the mvpp2 Ethernet controller driver, preparing the future addition of PPv2.2 support to the driver. The mvpp2 driver currently only supports PPv2.1, the Ethernet controller used on the Marvell Armada 375, and we are working on extending it to support PPv2.2, the Ethernet controller used on the Marvell Armada 7K/8K. PPv2.2 support is scheduled to be merged in 4.12.
  • Support for RaspberryPi platforms
    • Boris Brezillon contributed Device Tree changes to enable the VEC (Video Encoder) on all bcm283x platforms. Boris had previously contributed the driver for the VEC.

In addition to our direct contributions, a number of Bootlin engineers are also maintainers of various subsystems in the Linux kernel. As part of this maintenance role:

  • Maxime Ripard, co-maintainer of the Allwinner ARM platform, reviewed and merged 85 patches from contributors
  • Alexandre Belloni, maintainer of the RTC subsystem and co-maintainer of the Atmel ARM platform, reviewed and merged 60 patches from contributors
  • Grégory Clement, co-maintainer of the Marvell ARM platform, reviewed and merged 42 patches from contributors
  • Boris Brezillon, maintainer of the MTD NAND subsystem, reviewed and merged 8 patches from contributors

Here is the detailed list of contributions, commit per commit:

Bootlin and Raspberry Pi Linux kernel upstreaming

Raspberry Pi logoFor a few months, Bootlin has been helping the Raspberry Pi Foundation upstream to the Linux kernel a number of display related features for the Rasperry Pi platform.

The main goal behind this upstreaming process is to get rid of the closed-source firmware that is used on non-upstream kernels every time you need to enable/access a specific hardware feature, and replace it by something that is both open-source and compliant with upstream Linux standards.

Eric Anholt has been working hard to upstream display related features. His biggest contribution has certainly been the open-source driver for the VC4 GPU, but he also worked on the display controller side, and we were contracted to help him with this task.

Our first objective was to add support for SDTV (composite) output, which appeared to be much easier than we imagined. As some of you might already know, the display controller of the Raspberry Pi already has a driver in the DRM subsystem. Our job was to add support for the SDTV encoder (also called VEC, for Video EnCoder). The driver has been submitted just before the 4.10 merge window and surprisingly made it into 4.10 (see also the patches). Eric Anholt explained on his blog:

The Raspberry Pi Foundation recently started contracting with Bootlin to give me some support on the display side of the stack. Last week I got to review and release their first big piece of work: Boris Brezillon’s code for SDTV support. I had suggested that we use this as the first project because it should have been small and self contained. It ended up that we had some clock bugs Boris had to fix, and a bug in my core VC4 CRTC code, but he got a working patch series together shockingly quickly. He did one respin for a couple more fixes once I had tested it, and it’s now out on the list waiting for devicetree maintainer review. If nothing goes wrong, we should have composite out support in 4.11 (we’re probably a week late for 4.10).

Our second objective was to help Eric with HDMI audio support. The code has been submitted on the mailing list 2 weeks ago and will hopefully be queued for 4.12. This time on, we didn’t write much code, since Eric already did the bulk of the work. What we did though is debugging the implementation to make it work. Eric also explained on his blog:

Probably the biggest news of the last two weeks is that Boris’s native HDMI audio driver is now on the mailing list for review. I’m hoping that we can get this merged for 4.12 (4.10 is about to be released, so we’re too late for 4.11). We’ve tested stereo audio so far, no compresesd audio (though I think it should Just Work), and >2 channel audio should be relatively small amounts of work from here. The next step on HDMI audio is to write the alsalib configuration snippets necessary to hide the weird details of HDMI audio (stereo IEC958 frames required) so that sound playback works normally for all existing userspace, which Boris should have a bit of time to work on still.

On our side, it has been a great experience to work on such topics with Eric, and you should expect more contributions from Bootlin for the Raspberry Pi platform in the next months, so stay tuned!

Linux 4.9 released, Bootlin contributions

Linus Torvalds has released the 4.9 Linux kernel yesterday, as was expected. With 16214 non-merge commits, this is by far the busiest kernel development cycle ever, but in large part due to the merging of thousands of commits to add support for Greybus. LWN has very well summarized what’s new in this kernel release: 4.9 Merge window part 1, 4.9 Merge window part 2, The end of the 4.9 merge window.

As usual, we take this opportunity to look at the contributions Bootlin made to this kernel release. In total, we contributed 116 non-merge commits. Our most significant contributions this time have been:

  • Bootlin engineer Boris Brezillon, already a maintainer of the Linux kernel NAND subsystem, becomes a co-maintainer of the overall MTD subsystem.
  • Contribution of an input ADC resistor ladder driver, written by Alexandre Belloni. As explained in the commit log: common way of multiplexing buttons on a single input in cheap devices is to use a resistor ladder on an ADC. This driver supports that configuration by polling an ADC channel provided by IIO.
  • On Atmel platforms, improvements to clock handling, bug fix in the Atmel HLCDC display controller driver.
  • On Marvell EBU platforms
    • Addition of clock drivers for the Marvell Armada 3700 (Cortex-A53 based), by Grégory Clement
    • Several bug fixes and improvements to the Marvell CESA driver, for the crypto engine founds in most Marvell EBU processors. By Romain Perier and Thomas Petazzoni
    • Support for the PIC interrupt controller, used on the Marvell Armada 7K/8K SoCs, currently used for the PMU (Performance Monitoring Unit). By Thomas Petazzoni.
    • Enabling of Armada 8K devices, with support for the slave CP110 and the first Armada 8040 development board. By Thomas Petazzoni.
  • On Allwinner platforms
    • Addition of GPIO support to the AXP209 driver, which is used to control the PMIC used on most Allwinner designs. Done by Maxime Ripard.
    • Initial support for the Nextthing GR8 SoC. By Mylène Josserand and Maxime Ripard (pinctrl driver and Device Tree)
    • The improved sunxi-ng clock code, introduced in Linux 4.8, is now used for Allwinner A23 and A33. Done by Maxime Ripard.
    • Add support for the Allwinner A33 display controller, by re-using and extending the existing sun4i DRM/KMS driver. Done by Maxime Ripard.
    • Addition of bridge support in the sun4i DRM/KMS driver, as well as the code for a RGB to VGA bridge, used by the C.H.I.P VGA expansion board. By Maxime Ripard.
  • Numerous cleanups and improvements commits in the UBI subsystem, in preparation for merging the support for Multi-Level Cells NAND, from Boris Brezillon.
  • Improvements in the MTD subsystem, by Boris Brezillon:
    • Addition of mtd_pairing_scheme, a mechanism which allows to express the pairing of NAND pages in Multi-Level Cells NANDs.
    • Improvements in the selection of NAND timings.

In addition, a number of Bootlin engineers are also maintainers in the Linux kernel, so they review and merge patches from other developers, and send pull requests to other maintainers to get those patches integrated. This lead to the following activity:

  • Maxime Ripard, as the Allwinner co-maintainer, merged 78 patches from other developers.
  • Grégory Clement, as the Marvell EBU co-maintainer, merged 43 patches from other developers.
  • Alexandre Belloni, as the RTC maintainer and Atmel co-maintainer, merged 26 patches from other developers.
  • Boris Brezillon, as the MTD NAND maintainer, merged 24 patches from other developers.

The complete list of our contributions to this kernel release: