Category: Technical

Linux 4.18 released, Bootlin contributions

Penguin from Mylène Josserand
Drawing from Mylène Josserand,
based on a picture from Samuel Blanc under CC-BY-SA

With the 4.18 released last Sunday which needed an rc8 to mature, we take a bit of time to review our latest contributions.

As always, LWN.net did an interesting coverage of this release cycle merge window, highlighting the most important changes: the first half of the 4.18 merge window and the rest of the 4.18 merge window. For 4.18 only, Bootlin contributed a total of 190 patches, which puts us at the 13th place in the ranking of most contributing companies according to KPS.

Also according to LWN statistics, Bootlin’s engineer Alexandre Belloni is the 9th most active developer in terms of changed lines for this release with a total of 6801. We see the first contribution of Paul Kocialkowski, our intern working on Allwinner VPU driver, as a Bootlin team member. Finally, we’re proud to see the Linux kernel’s NAND subsystem welcoming Miquèl Raynal as a co-maintainer.

The main highlights of our contributions are:

Bootlin engineers are not only contributors, but also maintainers of various subsystems in the Linux kernel, which means they are involved in the process of reviewing, discussing and merging patches contributed to those subsystems:

  • Maxime Ripard, as the Allwinner platform co-maintainer, merged 38 patches from other contributors
  • Boris Brezillon, as the MTD/NAND maintainer, merged 76 patches from other contributors
  • Alexandre Belloni, as the RTC and Microsemi maintainer and Atmel platform co-maintainer, merged 32 patches from other contributors
  • Grégory Clement, as the Marvell EBU co-maintainer, merged 17 patches from other contributors

Here is the commit by commit detail of our contributions to 4.18:

Allwinner VPU support in mainline Linux status update (week 32)

This week started with the preparation of a new revision of the Cedrus VPU driver, after significant feedback was received on the version posted two weeks ago. Thanks to the careful testing carried out by community member Jernej Škrabec, a number of decoding issues were discovered in version 6 of the driver. This includes issues related to MPEG2 decoding but also to the use of the VPU untiling block, that affects all codecs indifferently.

The latest version of the driver (that was posted on Thursday) brings-in the required fixes for properly supporting the videos for which decoding errors were spotted.

Some updates were also included on the MPEG2 controls side, in order to bring them closer to the raw bitstream parameters. Some parameters (that are not exposed by VAAPI) were also added, making the V4L2 controls broader than what is strictly required for our VPU.

Regarding H265, progress was slow this week due to a mismatch between values provided by VAAPI and what our VPU expects. More specifically, VAAPI provides a byte-aligned value for the offset to the coded video data in the slice (which also includes a header with metadata) while our VPU expects a bit-aligned value that does not match the value provided by VAAPI. We are hard at work to figure out a solution to this issue, but it is not straightforward. In addition, the reference libvdpau-sunxi code does not set that offset explicitly, as it is reached after parsing the header through the VPU itself. In our case, the parsing is done in userspace so the use case differs.

Allwinner VPU support in mainline Linux status update (week 31)

Following on last week’s progress, this week was also focused on bringing the required plumbing for H265 support in our video decoding pipeline. Thanks to register dumps obtained last week from libvdpau-sunxi, it was possible to quickly hack together support for decoding a single intra frame (with no dependency on any other frame), by replaying the dumped register write sequence. Once decoding that single frame worked with the hardcoded register values, we progressively replaced these values with actual register field definitions, that have to be configured with the appropriate metadata for the frame, that is parsed from the H265 bitstream.

As a result, the next step was integrating the required metadata information as dedicated V4L2 controls. Since these controls have to be as generic as possible (in order to fit well with future V4L2 stateless VPU drivers), we carefully looked at the metadata fields that the bitstream offers and considered the elements that VAAPI provides in userspace as well as the information that our VPU needs specifically. It appears that some fields required by our VPU are not exposed by VAAPI directly, so a few tricks were needed along the way.

At this point, we have a first draft for the controls, that allow decoding the intra-coded frame that we dumped last week, but using the metadata provided through the controls instead of hardcoded values :

The very first H265 frame decoded with the Cedrus VPU driver! Can you guess which animation movie it is from?

More work is required to include support for other types of frame coding, namely B and P predictive frames. Next week’s focus will be set on decoding a series of intra-coded frames and moving on to supporting predictive frames. Thankfully, the work done by Bootlin engineer Maxime Ripard when adding support for H264 makes the whole process considerably easier, since H265 resembles H264 in many aspects.

Allwinner VPU support in mainline Linux status update (week 30)

This week’s progress in our VPU driver development effort was focused on two main tasks: submitting the sixth revision of the Cedrus VPU driver series to the mainline Linux kernel and starting the work on H265 decoding.

The patch series for this new iteration of the driver was submitted on Wednesday and contains both functional and cosmetic changes. Most notably, we implemented support for video-specific quantization matrices in MPEG2, one of the final extension bits we were missing until then, but also cleaned up the register definitions for the driver. At this point, there are no undocumented registers or fields left, which makes the overall understanding of the hardware interactions much more straightforward. The driver was also moved to staging drivers, not because it was deemed of poor quality but rather because V4L2 maintainers want to keep the ability to change the controls that our driver is using even after it is merged.

Aside of this work, we started looking into H265 decoding, that was also already implemented in libvdpau-sunxi for the downstream modified version of the Linux kernel provided by Allwinner for the H3 (still based on Linux 3.4 to this day, which was released in 2012). After setting up a board with this kernel and libvdpau-sunxi, we were able to dump the register access made by libvdpau-sunxi, providing a reference for bringing up H265 support in the Cedrus VPU driver!

Delivery of Allwinner VPU driver main goals

With a few weeks of delay, we are proud to announce the delivery of the main goals of our crowdfunding campaign dedicated at adding upstream Linux support for the Allwinner video decoding hardware.

After several months of hard work by Bootlin engineer Maxime Ripard and intern Paul Kocialkowski, we now have a working demo of Kodi running with our VPU driver on top of a mainline 4.18-rc kernel. Both MPEG2 and H264 are supported, with a fully-optimized pipeline between the VPU and the display side that does not involve any buffer copy or extra transformation that the hardware cannot offload. These results were possible thanks to the previous efforts carried out by the linux-sunxi community, and especially the libvdpau-sunxi project.

The Cedrus VPU driver running on the A33 and H3

Here were the main goals defined in our crowdfunding campaign, which we promised to deliver end of June 2018, and their status in our delivery:

  • Making sure that the codec works on the older Allwinner SoCs: A10, A13, A20, A33, R8 and R16.. This goal is fully met, with more features than planned: the Cedrus driver was brought up on the A10, A13, A20, A33 and H3. Therefore, we included H3 support in this delivery, even though it was originally only part of one of the stretch goals. The R8 is the same as an A13 and the R16 is the same as an A33, so they are supported as well.
  • Polishing the existing MPEG2 decoding support to make it fully production ready. This goal is fully met: we have done much more testing of the MPEG2 decoding, and both the Linux kernel code and user-space code supporting MPEG2 has been significantly improved and cleaned up.
  • Implementing H264 video decoding, since H264 is by far one of the most popular video codec.. This goal is fully met: H264 decoding support has been added to both the Linux kernel driver and the user-space library, including high-profile H264 support. However, the H264 support is still very recent and we expect that additional debugging and improvements will be needed.
  • Modifying the Allwinner display driver in order to be able to directly display the decoded frames instead of converting and copying those frames. This goal is fully met: the Allwinner DRM driver has received a number of patches to ensure we can use one of the several planes to directly display the video frames in the format provided by the VPU. Support for hardware scaling has also been fixed to work properly. Those patches have already been contributed to the upstream Linux kernel. The work on the A20 and A33 display driver was done by Bootlin, while the work on the H3 was done by other developers of the community.
  • Providing a user-space library easy to integrate in the popular open-source video players. This goal is partially met: while we are providing a libva-v4l2-request user-space libraries that can in theory be used by all libva capable video players, the actual integration with video players is for now only working completely with Kodi. We have started efforts to make it work with both VLC and GStreamer, but the work has not been complete due to various challenges detailed below. This area was definitely much more challenging than we initially expected.
  • Upstreaming those changes to the official Linux kernel. This goal is almost met: we have posted 5 iterations of the Cedrus Linux kernel driver, each time using new versions of the Request API patches, helping improve this API along the way. While our patches have not been merged yet, because the Request API itself hasn’t been merged, they have received significant review from the V4L developers, and we believe our patches are not far from being merged.

All in all, despite the numerous challenges encountered over the last few months, we are happy to see that we have been able to deliver most of the goals completely, and we are not too far off for the few goals that haven’t yet been fully met. As we will discuss below, we will continue to work in the next months on completing those unfinished steps, and on the stretch goals that received enough funding.

Reaching this level of support was not a straightforward journey, as our road was paved with various obstacles that are presented below.

Media Request API

In order to add support for the VPU found on Allwinner platforms, some internal plumbing is necessary in the Video4Linux2 (v4l2) framework, the video framework in Linux. While V4L2 gained support for a specific class of VPUs, so-called “stateful” (where the video bitstream is passed directly to the hardware controller) thanks to the Memory2Memory API, this is not sufficient for our hardware. Indeed, Allwinner platforms come with a “stateless” VPU, where the video needs to be parsed beforehand to extract the frame data and its associated metadata, and then passed to the hardware. V4L2 lacked an API for synchronizing the frame data and associated metadata, although it had been in development for a long time and known as the Request API.

Our work on Cedrus contributed to revive the flame for this API, that saw its development accelerated over the past months thanks to the commitment of individuals such as Alexandre Courbot, Hans Verkuil and Sakari Ailus. We had the opportunity to report various issues and suggest fixes over its development process, which were integrated so that all the required bits for our driver are now in. The API is finally mature and appears to be quite stable, so there is no known blocker left for its integration in the kernel.

Cedrus V4L2 Driver

The first version of the Cedrus driver originally developed in 2016 by Florent Revest as part of an internship at Bootlin was based on an old version of the Request API. We therefore started by porting it to the latest version of the API and kept publishing new revisions as development of the Request API happened. We also received useful feedback from the community in the process. Here are the different iterations of the Cedrus driver that have been sent as part of this crowdfunded effort:

In addition to those patch series adding the driver itself, an additional patch series was sent to bring H264 support.

The development of the driver itself was not the most cumbersome part of the process, although it brought some challenges. For instance, we had to rework buffer management after discovering a limitation in the hardware, where the luminance and chrominance planes of our destination buffers need to be kept close in memory. We also had to bring in a workqueue (later replaced by a threaded IRQ) for the needs of the M2M API, which comes with performance drawbacks, although this issue is in the process of being resolved.

Standalone Testing

In order to test the VPU driver in a fully-controlled environment, we developed a standalone testing tool: v4l2-request-test (formerly cedrus-frame-test) that implements all the V4L2 userspace APIs needed for our VPU, including M2M and the Request API. This tool includes frame data and metadata dumps from actual videos, with the ability to decode these frames one-by-one. The tool was tremendously helpful for debugging the driver as well as adding support for H264. Since the userspace APIs involved properly abstract the hardware, this tool can be used to bring up and develop other VPU drivers that rely on the V4L2 Request API!

VAAPI Backend

In order to provide integration with actual video players, we developed libva-v4l2-request (formerly libva-cedrus): a VAAPI backend that supports the V4L2 M2M and Request APIs. It currently supports both MPEG2 and H264 and will be extended as support for new formats is added. Just like v4l2-request-test, libva-v4l2-request aims at using the kernel APIs involved in a generic way, that should suit other Request API-based VPU drivers.

In the long run, it is likely that players will integrate direct support for the Request API (for instance, through ffmpeg). In the meantime, this allows interfacing with media players through two major interfaces: buffer derivation where the destination frames are copied (and converted to a regular image format when the VPU cannot do it on its own) or dma-buf, without any copy.

Zero-copy Pipeline Integration with EGL (Mali GPUs): VLC and GStreamer

In order to reach the best performance we can achieve, we focused on pipelines where no buffer copy is involved, on popular players: VLC and GStreamer. Since the X.org display server does not easily permit piping the VPU output to a dedicated plane on the Display Engine side, we investigated the use of the GPU. GPU support on Allwinner platforms still requires proprietary blobs at this point, such as the ones recently made available by Bootlin. We hope that the Lima project will soon bring a fully free alternative that will be integrated with both upstream kernel and upstream userspace components.

We did not have much luck when dealing with the tiled VPU output format, that the GPU cannot handle directly. Although we wrote a GPU shader for untiling (that works properly with regular GL implementations), the Mali GPU blobs did not behave as expected when it came to importing the tiled output frame. There is a chance that platforms that can output a regular image format (A33 and onwards) will be able to deal with piping the VPU and the GPU for accelerated scaling and colorspace conversion, but we did not test this option at this point.

Zero-copy Pipeline Integration with DRM (Display Engine): GStreamer and Kodi

Although involving the GPU in the pipeline was not a realistic possibility with the tiled VPU output format, various players support a direct DRM video output, that uses the Display Engine directly to pipe the video. Alas, it means that no window composition is possible, so this cannot be integrated with desktop environments. Instead, the players run standalone in their own virtual terminal.

We initially looked at using GStreamer this way but soon decided to prioritize Kodi (formerly XBMC), the popular mediacenter application. It was a struggle to integrate our pipeline (through libva-v4l2-request, via ffmpeg) in Kodi, although DRM video output support was there already. We eventually managed to get a usable result out of it, although there are areas left to improve!

LibreELEC Image Release with Kodi

In order to showcase the delivery of our main VPU crowdfunding campaign goals, we cooked a release of LibreELEC that supports the A20, A33 and H3 SoCs! It consists of a LibreELEC root filesystem (excluding the kernel and boot software) that works in conjunction with our latest linux-cedrus kernel tree.

Source code is of course available through our repositories, marked with the release-2018-07 tag.
Instructions to deploy the software on a compatible board are available on the linux-sunxi community wiki!

Remaining Tasks

We have tackled many of the tasks on our plate at this point, but there are still items that need to be worked on:

  • posting new series of the Cedrus driver and H264 support until it is merged;
  • supporting H265 in our driver and userspace components;
  • supporting the ARM64 SoCs that come with version 2 of the Display Engine design, namely the H5 and A64;
  • contributing to the integration of our code in upstream Kodi and LibreELEC;
  • integrating a dma-buf and DRM pipeline with GStreamer.

Thanks

We would like to thank all the individuals and companies who have supported this project by participating to our crowdfunding campaign, but also the linux-sunxi community members who did the initial reverse engineering of the Cedrus VPU and who worked with us during the development of this driver as well as the members of the V4L2 community who worked on the Request API and reviewed our patches.

Allwinner VPU support in mainline Linux status update (week 28)

This week was the occasion to send out version 5 of the Sunxi-Cedrus VPU driver, that uses version 16 of the media requests API. The API contains the necessary internal plumbing for tying specific metadata (exposed as v4l2 controls, that are structures of data set by userspace) about the current video frame to decode with the associated source buffer (that is extracted in slices from the raw video bitstream and contains the frame’s encoded data). Adding this feature to the Linux kernel paves the way for supporting stateless VPUs such as Allwinner’s Video Engine, that are found in various ARM platforms. With version 16, a number of reliability issues were fixed and we were able to run decoding tests for hours without hitting any error!

This new version of our driver contains several improvements, that are presented in the cover letter of the series. Most notably, it brings support for the H3 (which uses the second revision of the Allwinner’s Display Engine hardware block) and exposes linear YUV output in addition to the tiled output format. The issue related to H264 decoding failing because of the luma and chroma planes being too distant in memory was fixed by allocating contiguous buffers for the destination frames. However, this required significant changes in our display pipeline, which was the occasion to rework both cedrus-frame-test and libva-cedrus to handle various scenarios for buffer and planes matching and avoid hardcoded values that are specific to our pipeline. This opens the way to making these tools generic users of the V4L2 and DRM APIs, without any particular tie to our specific platform and setup.

H264 support (for the baseline profile) was also merged in our cedrus kernel tree, as well as cedrus-frame-test and libva-cedrus master branches. With that, it was possible to cook a LibreELEC build with support for our updated libva-cedrus, that allows decoding H264 videos! The result is presented in the video below, that runs on an A33 and shows Sintel, an animation movie made with free software by the Blender Foundation and released under the Creative Commons Attribution 3.0 Generic license:

We also spent some time figuring out the reason for the various artifacts found on the A20 when using the display scaler. It turned out to be some missing register, and one register where the value documented would be offset by one, resulting in the last line of the picture repeating itself.

Once done, we switched to working on the issue we mentionned last week with H264. After testing a few ideas, we now have the H264 high profile working with libva-dump and cedrus-frame-test. The next step will be to port the new code to handle the reference frames to libva-cedrus, and hopefully we will be able to have this in our usual players.

Allwinner VPU support in mainline Linux status update (week 27)

This week, significant time was dedicated to preparing a new revision of the Sunxi-Cedrus VPU kernel driver. This new version (that was started last week) based on version 15 of the media requests API brought about a number of challenges. First off, integrating the recently-tested VPU-side untiling of the destination buffers required a significant rewrite of the part in charge of managing formats and buffers. The part of our driver handling V4L2 controls (that are used to submit the frame metadata) was also significantly reworked to allow validating that the frame metadata has indeed been submitted by userspace before launching a decode run. An initial implementation of this was brought up and discussed with V4L2 maintainer Hans Verkuil, who is backing (and baking) the requests API series. He came up with a specific patch that should allow properly implementing this detection at the right time (when checking the media request’s validity, instead of at the start of the run). Hans also solved various reliability issues that we were experiencing when using the requests API with our driver. As a result, he posted version 16 of the requests API series with these fixes. We are hoping that this version will be one of the final iterations of this long-awaited series!

While rebasing H264 support, we experienced a strange issue where the destination buffers were sometimes corrupted and sometimes not. All the hardware configuration (register writes) were exactly the same, except for the buffer addresses (that naturally tend to change depending on allocations order in the related CMA memory pool). After some investigation, we discovered that when the gap between the luma and chroma planes of the destination buffer are too distant, a corruption happens. It may be that some offset is used in the hardware at some point and that it is not coded on enough bits to represent a large gap. The way to work around this is to make sure that all the planes of our destination buffer are allocated contiguously. In practice, this means that we need a single allocation for the each whole destination buffer (with the size of its two planes), ensuring that there is no gap between the planes.

A random bug in the development of Sunxi-Cedrus with funky colors

The work has continued on H264, and especially to add support for the High Profile decoding. My test video showed a limitation in our current code however, due to what seems to be a limitation of the libva API. Indeed, the H264 codec relies on a decoded picture buffer (DPB) that holds the previous decoded pictures that might be used as reference frames to decode the current frame. The kernel interface needs that DPB, and our driver will also need it to perform some ID assignation for the current frame. However, libva only gives the list of frames needed to decode the current frame, and not the whole DPB. That leads to a situation where subsequent frames, using the same reference frames set, will be assigned the same ID, which obviously doesn’t work very well. Most of the week has been spent trying to evaluate how we can address that issue, and to start implementing a solution that would be based on a cache of the reference frames passed to our libva driver.

Allwinner VPU support in mainline Linux status update (week 26)

This week on the video player integration side, we focused on the last remaining bits for Kodi integration with our pipeline, with support for the H3 SoC. Unlike its predecessors, the untiling is done directly by the VPU itself and not by the display engine, which makes the overall integration easier. Allwinner platforms that come with version 2 of the display engine only support linear YUV formats, that is piped directly from the VPU. This makes our pipeline for these platforms slightly different from previous generations, since the VPU needs to be configured properly and no DRM format modifier needs to be carried around on the software side.

Thankfully, the code for enabling this output was already implemented by the linux-sunxi community in libvdpau-sunxi and only minor changes had to be added, in order to use the NV12 YUV format for the luma and chroma planes. Thanks to the help of the LibreELEC community members interested in Allwinner devices, who are as always very helpful when it comes to working with Kodi on Allwinner platforms, it was possible to pipe everything together on the H3 (and the DE2 DRM driver even received an initial patch for z-pos support, that allows re-ordering layers for alpha blending).

The result is visible in the following video, that shows Elephants Dream, an animation movie made with free software by the Blender Foundation and released under the Creative Commons Attribution 2.5 Generic license:

On the V4L2 kernel driver side, a rebase of the Sunxi-Cedrus driver on top of the latest kernel release candidate, 4.18-rc2 was completed and a number of features are being included in the series, including H3 support and support for the untiled NV12 video output. We expect to send a new iteration of the kernel driver patches next week.

We also discovered that the VPU untiling block is available on the A33 too, which will make supporting the A33 easier as well.

Finally, there was no progress on the H264 front this week, as Maxime was away on a business trip teaching a training. The progress on H264 will resume next week.

More OpenGL binaries for the Mali support on Allwinner platforms with mainline Linux

Allwinner SoCBack in September, we announced the availability of Mali userspace blobs that provide OpenGL acceleration on Allwinner platforms using the mainline Linux kernel. Back then, only the r6p2 version of the Mali blobs were available, with only the x11 and fbdev backends, and only for ARM 32 bits. Following the announcement we made last September, we kept talking with Allwinner to release more binaries and increase the usefulness of them. Two major categories were missing in order to complete the previous batch of binaries Allwinner allowed us to distribute: Wayland and arm64 flavours.

After some discussions, Allwinner provided to us this week additional Mali blobs, covering Wayland support, ARM64, and also newer versions for some of them. Overall, we now provide:

  • r6p2 version, ARM 32 bits, X11
  • r6p2 version, ARM 32 bits, fbdev
  • r6p2 version, ARM 32 bits, Wayland (new)
  • r6p2 version, ARM 64 bits, X11 (new)
  • r6p2 version, ARM 64 bits, fbdev (new)
  • r6p2 version, ARM 64 bits, Wayland (new)
  • r8p1 version, ARM 32 bits, fbdev (new)
  • r8p1 version, ARM 64 bits, fbdev (new)

We pushed everything to our github repo, enjoy! See our previous blog post for instructions on how to use those blobs.

Those binary blobs are useful because they allow today to have a fully working OpenGL acceleration on Allwinner platforms: we recently ran a Qt5 application doing OpenGL rendering 24/7 on an Allwinner A33 platform for 1.5 month uninterrupted, as a stability test. Of course, long term, we are following the progress of the Lima project, which will provide a completely free and open-source solution to provide OpenGL acceleration on Allwinner platforms.

Bleeding edge toolchains updated

Since last year, our site toolchains.bootlin.com provides a large selection of ready-to-use cross-compilation toolchains, covering a wide range of CPU architectures and C libraries. We have just deployed today a new update to all our bleeding-edge toolchains. Those toolchains are now based on:

  • GCC 8.1.0
  • Binutils 2.30
  • glibc 2.27 (plus fixes) or uClibc-ng 1.0.30 or musl 1.1.19
  • Linux headers 4.14
  • GDB 8.1

Toolchains.bootlin.com

All our 77 bleeding-edge toolchains built successfully with those component versions, and many of them received runtime testing under Qemu. We would like to do a special thanks to Romain Naour from Smile, who contributed a lot to this update by adding GCC 8.1 and GDB 8.1 support in Buildroot, and fixing a number of issues discovered when building those toolchains.

We will continue to regularly update our toolchains, and we are very interested in receiving feedback about those toolchains, to fix any issue or extend the range of configurations that are covered. Do not hesitate to get in touch!