Crowdfunding campaign for upstream Linux kernel driver for Allwinner VPU

Back in 2012, Bootlin (formerly Free Electrons) engineer Maxime Ripard pioneered the support for Allwinner processors in the official Linux kernel. Today, thanks to the contributions of numerous developers around the world and our involvement, there is very good support for a large number of Allwinner processors in the Linux kernel, to the point where actual Allwinner-based products are shipping with the mainline kernel.

Despite this major effort, there is one area that has remained unsupported in the mainline kernel: the video decoding and encoding engine, which allows to accelerate in hardware the decoding and encoding of popular codecs such as MPEG2, MPEG4 or H264. Last summer, we successfully implemented a prototype, supporting MPEG2 decoding and partially MPEG4 decoding.

Today, we are launching a crowdfunding campaign to fund the remainder of the development: finishing MPEG4 decoding support, implementing H264 decoding, optimizing the rendering of video frames in cooperation with the display driver, and upstreaming the driver. We also have additional goals of supporting H265, encoding support, and additional Allwinner SoCs.

In the vendor-provided kernel, this video decoding/encoding unit is supported by a kernel driver that uses a non-standard user-space API, in conjunction with a binary-only userspace blob. Fortunately, a number of people have done an enormous reverse engineering effort, which we have leveraged for our existing prototype, and which we intend to use to continue the development of this upstream driver. Both Maxime Ripard and our intern Paul Kocialkowski will be working on this crowdfunded project.

This is our first crowdfunding campaign to fund upstream Linux kernel development, and we are interested in seeing how much interest there is in such a financing model. Help us making this a success by spreading the word!

Mali OpenGL support on Allwinner platforms with mainline Linux

As most people know, getting GPU-based 3D acceleration to work on ARM platforms has always been difficult, due to the closed nature of the support for such GPUs. Most vendors provide closed-source binary-only OpenGL implementations in the form of binary blobs, whose quality depend on the vendor.

This situation is getting better and better through vendor-funded initiatives like for the Broadcom VC4 and VC5, or through reverse engineering projects like Nouveau on Tegra SoCs, Etnaviv on Vivante GPUs, Freedreno on Qualcomm’s. However there are still GPUs where you do not have the option to use a free software stack: PowerVR from Imagination Technologies and Mali from ARM (even though there is some progress on the reverse engineering effort).

Allwinner SoCs are using either a Mali GPU from ARM or a PowerVR from Imagination Technologies, and therefore, support for OpenGL on those platforms using a mainline Linux kernel has always been a problem. This is also further complicated by the fact that Allwinner is mostly interested in Android, which uses a different C library that avoids its use in traditional glibc-based systems (or through the use of libhybris).

However, we are happy to announce that Allwinner gave us clearance to publish the userspace binary blobs that allows to get OpenGL supported on Allwinner platforms that use a Mali GPU from ARM, using a recent mainline Linux kernel. Of course, those are closed source binary blobs and not a nice fully open-source solution, but it nonetheless allows everyone to have OpenGL support working, while taking advantage of all the benefits of a recent mainline Linux kernel. We have successfully used those binary blobs on customer projects involving the Allwinner A33 SoCs, and they should work on all Allwinner SoCs using the Mali GPU.

In order to get GPU support to work on your Allwinner platform, you will need:

  • The kernel-side driver, available on Maxime Ripard’s Github repository. This is essentially the Mali kernel-side driver from ARM, plus a number of build and bug fixes to make it work with recent mainline Linux kernels.
  • The Device Tree description of the GPU. We introduced Device Tree bindings for Mali GPUs in the mainline kernel a while ago, so that Device Trees can describe such GPUs. Such description has been added for the Allwinner A23 and A33 SoCs as part of this commit.
  • The userspace blob, which is available on Free Electrons GitHub repository. It currently provides the r6p2 version of the driver, with support for both fbdev and X11 systems. Hopefully, we’ll gain access to newer versions in the future, with additional features (such as GBM support).

If you want to use it in your system, the first step is to have the GPU definition in your device tree if it’s not already there. Then, you need to compile the kernel module:

git clone https://github.com/mripard/sunxi-mali.git
cd sunxi-mali
export CROSS_COMPILE=$TOOLCHAIN_PREFIX
export KDIR=$KERNEL_BUILD_DIR
export INSTALL_MOD_PATH=$TARGET_DIR
./build.sh -r r6p2 -b
./build.sh -r r6p2 -i

It should install the mali.ko Linux kernel module into the target filesystem.

Now, you can copy the OpenGL userspace blobs that match your setup, most likely the fbdev or X11-dma-buf variant. For example, for fbdev:

git clone https://github.com/free-electrons/mali-blobs.git
cd mali-blobs
cp -a r6p2/fbdev/lib/lib_fb_dev/lib* $TARGET_DIR/usr/lib

You should be all set. Of course, you will have to link your OpenGL applications or libraries against those user-space blobs. You can check that everything works using OpenGL test programs such as es2_gears for example.

Linux 4.9 released, Free Electrons contributions

Linus Torvalds has released the 4.9 Linux kernel yesterday, as was expected. With 16214 non-merge commits, this is by far the busiest kernel development cycle ever, but in large part due to the merging of thousands of commits to add support for Greybus. LWN has very well summarized what’s new in this kernel release: 4.9 Merge window part 1, 4.9 Merge window part 2, The end of the 4.9 merge window.

As usual, we take this opportunity to look at the contributions Free Electrons made to this kernel release. In total, we contributed 116 non-merge commits. Our most significant contributions this time have been:

  • Free Electrons engineer Boris Brezillon, already a maintainer of the Linux kernel NAND subsystem, becomes a co-maintainer of the overall MTD subsystem.
  • Contribution of an input ADC resistor ladder driver, written by Alexandre Belloni. As explained in the commit log: common way of multiplexing buttons on a single input in cheap devices is to use a resistor ladder on an ADC. This driver supports that configuration by polling an ADC channel provided by IIO.
  • On Atmel platforms, improvements to clock handling, bug fix in the Atmel HLCDC display controller driver.
  • On Marvell EBU platforms
    • Addition of clock drivers for the Marvell Armada 3700 (Cortex-A53 based), by Grégory Clement
    • Several bug fixes and improvements to the Marvell CESA driver, for the crypto engine founds in most Marvell EBU processors. By Romain Perier and Thomas Petazzoni
    • Support for the PIC interrupt controller, used on the Marvell Armada 7K/8K SoCs, currently used for the PMU (Performance Monitoring Unit). By Thomas Petazzoni.
    • Enabling of Armada 8K devices, with support for the slave CP110 and the first Armada 8040 development board. By Thomas Petazzoni.
  • On Allwinner platforms
    • Addition of GPIO support to the AXP209 driver, which is used to control the PMIC used on most Allwinner designs. Done by Maxime Ripard.
    • Initial support for the Nextthing GR8 SoC. By Mylène Josserand and Maxime Ripard (pinctrl driver and Device Tree)
    • The improved sunxi-ng clock code, introduced in Linux 4.8, is now used for Allwinner A23 and A33. Done by Maxime Ripard.
    • Add support for the Allwinner A33 display controller, by re-using and extending the existing sun4i DRM/KMS driver. Done by Maxime Ripard.
    • Addition of bridge support in the sun4i DRM/KMS driver, as well as the code for a RGB to VGA bridge, used by the C.H.I.P VGA expansion board. By Maxime Ripard.
  • Numerous cleanups and improvements commits in the UBI subsystem, in preparation for merging the support for Multi-Level Cells NAND, from Boris Brezillon.
  • Improvements in the MTD subsystem, by Boris Brezillon:
    • Addition of mtd_pairing_scheme, a mechanism which allows to express the pairing of NAND pages in Multi-Level Cells NANDs.
    • Improvements in the selection of NAND timings.

In addition, a number of Free Electrons engineers are also maintainers in the Linux kernel, so they review and merge patches from other developers, and send pull requests to other maintainers to get those patches integrated. This lead to the following activity:

  • Maxime Ripard, as the Allwinner co-maintainer, merged 78 patches from other developers.
  • Grégory Clement, as the Marvell EBU co-maintainer, merged 43 patches from other developers.
  • Alexandre Belloni, as the RTC maintainer and Atmel co-maintainer, merged 26 patches from other developers.
  • Boris Brezillon, as the MTD NAND maintainer, merged 24 patches from other developers.

The complete list of our contributions to this kernel release:

Linux 4.8 released, Free Electrons contributions

Adelie PenguinLinux 4.8 has been released on Sunday by Linus Torvalds, with numerous new features and improvements that have been described in details on LWN: part 1, part 2 and part 3. KernelNewbies also has an updated page on the 4.8 release. We contributed a total of 153 patches to this release. LWN also published some statistics about this development cycle.

Our most significant contributions:

  • Boris Brezillon improved the Rockchip PWM driver to avoid glitches basing that work on his previous improvement to the PWM subsystem already merged in the kernel. He also fixed a few issues and shortcomings in the pwm regulator driver. This is finishing his work on the Rockchip based Chromebook platforms where a PWM is used for a regulator.
  • While working on the driver for the sii902x HDMI transceiver, Boris Brezillon did a cleanup of many DRM drivers. Those drivers were open coding the encoder selection. This is now done in the core DRM subsystem.
  • On the support of Atmel platforms
    • Alexandre Belloni cleaned up the existing board device trees, removing unused clock definitions and starting to remove warnings when compiling with the Device Tree Compiler (dtc).
  • On the support of Allwinner platforms
    • Maxime Ripard contributed a brand new infrastructure, named sunxi-ng, to manage the clocks of the Allwinner platforms, fixing shortcomings of the Device Tree representation used by the existing implementation. He moved the support of the Allwinner H3 clocks to this new infrastructure.
    • Maxime also developed a driver for the Allwinner A10 Digital Audio controller, bringing audio support to this platform.
    • Boris Brezillon improved the Allwinner NAND controller driver to support DMA assisted operations, which brings a very nice speed-up to throughput on platforms using NAND flashes as the storage, which is the case of Nextthing’s C.H.I.P.
    • Quentin Schulz added support for the Allwinner R16 EVB (Parrot) board.
  • On the support of Marvell platforms
    • Grégory Clément added multiple clock definitions for the Armada 37xx series of SoCs.
    • He also corrected a few issues with the I/O coherency on some Marvell SoCs
    • Romain Perier worked on the Marvell CESA cryptography driver, bringing significant performance improvements, especially for dmcrypt usage. This driver is used on numerous Marvell platforms: Orion, Kirkwood, Armada 370, XP, 375 and 38x.
    • Thomas Petazzoni submitted a driver for the Aardvark PCI host controller present in the Armada 3700, enabling PCI support for this platform.
    • Thomas also added a driver for the new XOR engine found in the Armada 7K and Armada 8K families

Here are in details, the different contributions we made to this release:

Support for the Allwinner VPU in the mainline Linux kernel

Over the last few years, and most recently with the support for the C.H.I.P platform, Free Electrons has been heavily involved in initiating and improving the support in the mainline Linux kernel for the Allwinner ARM processors. As of today, a large number of hardware features of the Allwinner processors, especially the older ones such as the A10 or the A13 used in the CHIP, are usable with the mainline Linux kernel, including complex functionality such as display support and 3D acceleration. However, one feature that was still lacking is proper support for the Video Processing Unit (VPU) that allows to accelerate in hardware the decoding and encoding of popular video formats.

During the past two months, Florent Revest, a 19 year old intern at Free Electrons worked on a mainline solution for this Video Processing Unit. His work followed the reverse engineering effort of the Cedrus project, and this topic was also listed as a High Priority Reverse Engineering Project by the FSF.

The internship resulted in a new sunxi-cedrus driver, a Video4Linux memory-to-memory decoder kernel driver and a corresponding VA-API backend, which allows numerous userspace applications to use the decoding capabilities. Both projects have both been published on Github:

Currently, the combination of the kernel driver and VA-API backend supports MPEG2 and MPEG4 decoding only. There is for the moment no support for encoding, and no support for H264, though we believe support for both aspects can be added within the architecture of the existing driver and VA-API backend.

A first RFC patchset of the kernel driver has been sent to the linux-media mailing list, and a complete documentation providing installation information and architecture details has been written on the linux-sunxi’s wiki.

Here is a video of VLC playing a MPEG2 demo video on top of this stack on the Next Thing’s C.H.I.P:

Free Electrons working on the $9 C.H.I.P. computer

C.H.I.P computer

If you’re following the news about embedded Linux and new cool development platforms, you for sure couldn’t miss the announcement on the world’s first $9 computer. This computer, called C.H.I.P., was started through a crowd-funding campaign on Kickstarter and reception in the Free Software and Open Source community has been very positive. Out of an initial funding goal of $50,000, NextThing Co, the Oakland, California based company creating this product eventually managed to raise more $2,000,000 of funding.

NextThing Co announced their intention to support the platform in the most open way possible: the schematics will be made available, and it will be supported in the mainline Linux kernel.

It turns out that the processor NextThing Co has chosen for this platform is an Allwinner R8 processor. Free Electrons has been working since several years on supporting Allwinner processors in the mainline Linux kernel: our engineer Maxime Ripard is the maintainer of the Allwinner SoC support.

Thanks to this long term involvement, Free Electrons has been asked by NextThing Co to work with them to support the C.H.I.P. computer in the mainline Linux kernel, and in the process bring some significant improvements to the support of Allwinner processors in the kernel.

C.H.I.P, world first $9 computer

NextThing Co announced recently our collaboration in a blog post on Kickstarter:

We’re incredibly excited to announce that we’ve partnered with one of the premier contributors to ARM Linux: Free Electrons! We will be collaborating with their amazing team of ARM Linux engineers, and of course our Kernel Hacker backers to help us test and mainline C.H.I.P.’s kernel modifications as we move forward.

Free Electrons is also very excited to be working with NextThing Co on this project! Thanks to this, over the next months, we will have a very substantial amount of time dedicated to this project, and we will regularly push code to support the missing hardware Allwinner SoC hardware blocks in the mainline Linux kernel and to support the C.H.I.P. board.

Pocket C.H.I.P

More details about the C.H.I.P:

  • Availability planned for 2016 for the general public. A selection of 1000 kernel hackers who backed on Kickstarter will have the platform earlier.
  • Very small platform: 40mm x 60mm, making is even smaller than a Raspberry Pi or BeagleBone.
  • Allwinner R8 processor, clocked at 1 Ghz, and offering OpenGL/OpenVG acceleration through an ARM Mali GPU
  • 512 MB of RAM
  • 4GB of on-board NAND flash storage
  • WiFi and Bluetooth 4.0 to connect the system to the outside world
  • One USB host port
  • Powered through a micro USB connector which also supports USB OTG (either USB host or device).
  • Jack connector for composite video out, headphones and microphone input.
  • Many headers to connect external devices (SPI, I2C, UART + 8 GPIOs)
  • Integrated circuit for charging a LiPo battery and being powered by it
  • Additional HDMI or VGA add-on boards will be needed to connect to displays with the corresponding connectivity.