Tag: kickstarter

This week was the occasion to send out version 5 of the Sunxi-Cedrus VPU driver, that uses version 16 of the media requests API. The API contains the necessary internal plumbing for tying specific metadata (exposed as v4l2 controls, that are structures of data set by userspace) about the current video frame to decode with the associated source buffer (that is extracted in slices from the raw video bitstream and contains the frame’s encoded data). Adding this feature to the Linux kernel paves the way for supporting stateless VPUs such as Allwinner’s Video Engine, that are found in various ARM platforms. With version 16, a number of reliability issues were fixed and we were able to run decoding tests for hours without hitting any error!

This new version of our driver contains several improvements, that are presented in the cover letter of the series. Most notably, it brings support for the H3 (which uses the second revision of the Allwinner’s Display Engine hardware block) and exposes linear YUV output in addition to the tiled output format. The issue related to H264 decoding failing because of the luma and chroma planes being too distant in memory was fixed by allocating contiguous buffers for the destination frames. However, this required significant changes in our display pipeline, which was the occasion to rework both cedrus-frame-test and libva-cedrus to handle various scenarios for buffer and planes matching and avoid hardcoded values that are specific to our pipeline. This opens the way to making these tools generic users of the V4L2 and DRM APIs, without any particular tie to our specific platform and setup.

H264 support (for the baseline profile) was also merged in our cedrus kernel tree, as well as cedrus-frame-test and libva-cedrus master branches. With that, it was possible to cook a LibreELEC build with support for our updated libva-cedrus, that allows decoding H264 videos! The result is presented in the video below, that runs on an A33 and shows Sintel, an animation movie made with free software by the Blender Foundation and released under the Creative Commons Attribution 3.0 Generic license:

We also spent some time figuring out the reason for the various artifacts found on the A20 when using the display scaler. It turned out to be some missing register, and one register where the value documented would be offset by one, resulting in the last line of the picture repeating itself.

Once done, we switched to working on the issue we mentionned last week with H264. After testing a few ideas, we now have the H264 high profile working with libva-dump and cedrus-frame-test. The next step will be to port the new code to handle the reference frames to libva-cedrus, and hopefully we will be able to have this in our usual players.

This week, significant time was dedicated to preparing a new revision of the Sunxi-Cedrus VPU kernel driver. This new version (that was started last week) based on version 15 of the media requests API brought about a number of challenges. First off, integrating the recently-tested VPU-side untiling of the destination buffers required a significant rewrite of the part in charge of managing formats and buffers. The part of our driver handling V4L2 controls (that are used to submit the frame metadata) was also significantly reworked to allow validating that the frame metadata has indeed been submitted by userspace before launching a decode run. An initial implementation of this was brought up and discussed with V4L2 maintainer Hans Verkuil, who is backing (and baking) the requests API series. He came up with a specific patch that should allow properly implementing this detection at the right time (when checking the media request’s validity, instead of at the start of the run). Hans also solved various reliability issues that we were experiencing when using the requests API with our driver. As a result, he posted version 16 of the requests API series with these fixes. We are hoping that this version will be one of the final iterations of this long-awaited series!

While rebasing H264 support, we experienced a strange issue where the destination buffers were sometimes corrupted and sometimes not. All the hardware configuration (register writes) were exactly the same, except for the buffer addresses (that naturally tend to change depending on allocations order in the related CMA memory pool). After some investigation, we discovered that when the gap between the luma and chroma planes of the destination buffer are too distant, a corruption happens. It may be that some offset is used in the hardware at some point and that it is not coded on enough bits to represent a large gap. The way to work around this is to make sure that all the planes of our destination buffer are allocated contiguously. In practice, this means that we need a single allocation for the each whole destination buffer (with the size of its two planes), ensuring that there is no gap between the planes.

The work has continued on H264, and especially to add support for the High Profile decoding. My test video showed a limitation in our current code however, due to what seems to be a limitation of the libva API. Indeed, the H264 codec relies on a decoded picture buffer (DPB) that holds the previous decoded pictures that might be used as reference frames to decode the current frame. The kernel interface needs that DPB, and our driver will also need it to perform some ID assignation for the current frame. However, libva only gives the list of frames needed to decode the current frame, and not the whole DPB. That leads to a situation where subsequent frames, using the same reference frames set, will be assigned the same ID, which obviously doesn’t work very well. Most of the week has been spent trying to evaluate how we can address that issue, and to start implementing a solution that would be based on a cache of the reference frames passed to our libva driver.

This week on the video player integration side, we focused on the last remaining bits for Kodi integration with our pipeline, with support for the H3 SoC. Unlike its predecessors, the untiling is done directly by the VPU itself and not by the display engine, which makes the overall integration easier. Allwinner platforms that come with version 2 of the display engine only support linear YUV formats, that is piped directly from the VPU. This makes our pipeline for these platforms slightly different from previous generations, since the VPU needs to be configured properly and no DRM format modifier needs to be carried around on the software side.

Thankfully, the code for enabling this output was already implemented by the linux-sunxi community in libvdpau-sunxi and only minor changes had to be added, in order to use the NV12 YUV format for the luma and chroma planes. Thanks to the help of the LibreELEC community members interested in Allwinner devices, who are as always very helpful when it comes to working with Kodi on Allwinner platforms, it was possible to pipe everything together on the H3 (and the DE2 DRM driver even received an initial patch for z-pos support, that allows re-ordering layers for alpha blending).

The result is visible in the following video, that shows Elephants Dream, an animation movie made with free software by the Blender Foundation and released under the Creative Commons Attribution 2.5 Generic license:

On the V4L2 kernel driver side, a rebase of the Sunxi-Cedrus driver on top of the latest kernel release candidate, 4.18-rc2 was completed and a number of features are being included in the series, including H3 support and support for the untiled NV12 video output. We expect to send a new iteration of the kernel driver patches next week.

We also discovered that the VPU untiling block is available on the A33 too, which will make supporting the A33 easier as well.

Finally, there was no progress on the H264 front this week, as Maxime was away on a business trip teaching a training. The progress on H264 will resume next week.

This week started off by submitting the fourth revision of the Sunxi-Cedrus VPU driver for review. Many improvements were squashed into this new version and the driver is closer than ever to being merged. With the media requests API in a nearly-ready state, things are really coming together on the kernel side.

On the userspace side, our standalone testing tool cedrus-frame-test received a number of improvements, starting with Maxime’s H264 work that was rebased and integrated in the master branch. Atomic modesetting support with DRM planes was also completed and merged. It allowed completing dma-buf support in the tool, implementing a zero-copy pipeline. With asynchronous page-flipping, performance is getting real good with only a few milliseconds required to schedule the flip and no buffer duplication involved!

Regarding integration with Kodi, we moved forward with the code using ffmpeg’s hwcontext for decoding with VAAPI and mapping from the VAAPI output to DRM, ending up through Kodi’s DRMMPrimeRenderer. The display pipeline is pretty much the same as cedrus-frame-test with atomic modesetting and dma-buf, only that Kodi uses an extra plane on top for displaying its controls and interface.

There are still some configuration issues to work on for display (and perhaps some kind of corruption happening on the display engine’s side), as illustrated on the following picture:

This week has seen some good H264 progress too! Our libva implementation has been tested, and while we encountered some VLC bugs that makes it drop the first few seconds, once passed that bug, every frame is decoded properly using a baseline profile H264 video. We’ve discussed with VLC developpers about this, and since it also affects the H264 software decoding, we will probably turn this into a bug report (and hopefully a bug fix!).

We therefore started to work on implementing the high profile support. We went back to the method we were using when first developping the baseline profile support: we dumped the registers access of the libvdpau-sunxi decoding the video on an Allwinner 3.4 kernel, and comparing the registers accesses we were doing. This is very early at this stage, so we don’t have much to show for now, but stay tuned for more news!

Integration with video players

Following up on last week’s efforts on the video players integration front, Kodi remained our core focus. With a LibreELEC setup in place, it was possible to start tackling VAAPI integration. This was not such a straightforward task, since various assumptions were in place. For instance, it was assumed that VAAPI support was only relevant for x86 platforms and it seems pretty clear that VAAPI integration in general was done with x86 in mind. This is particularly illustrated by the fact that the VAAPI video rendering pipeline relies on the GPU for all transformations and composition. This is a typical setup for x86, as the use of planes on these platforms was progressively replaced by a GPU-centric approach. Since our goal with Kodi is to use DRM/KMS planes in place of the GPU, this did not fit well. Moreover, the GPU import format required for dma-buf is simply not supported by the Mali blob (as we found out some weeks ago when working with VLC and the GLES untiler) and this is the only setup that Kodi currently supports for VAAPI.

There is still definitely hope, as Kodi supports a DRM Prime renderer that uses DRM/KMS planes in place of the GPU but does not support VAAPI in its current form. More specifically, it uses ffmpeg to get a dma-buf handle (through the AV_PIX_FMT_DRM_PRIME format from ffmpeg), that is not available as-is. In order to get this sort of pipeline with VAAPI, multiple steps have to be taken. A hardware acceleration context has to be brought up to select the VAAPI acceleration method instead of regular software decoding. This exposes the AV_PIX_FMT_VAAPI format from ffmpeg, which is still not good to feed the Kodi DRM Prime renderer. This has to be converted to AV_PIX_FMT_DRM_PRIME using ffmpeg helpers. As a result, some plumbing is required in Kodi and this work is still work in progress at the moment.

In parallel to the work on players, our Sunxi-Cedrus VPU driver was rebased on top of the latest version of the media request API from Hans Verkuil. It was the occasion to spot various bugs in this latest iteration, that were rapidly tackled thanks to Hans’ availability. The required follow-up patches were posted on the request API branch and will be part of its next revision. Regarding our driver itself, a great number of comments from our previous patchset were taken into account and integrated. We now have another iteration of the series ready, that we will publish soon. The tasks list for the driver itself keeps shrinking and we are getting closer and closer to the point where the driver is ready to be merged!

H264 support

On the H264 front, good progress has been made this week too. Early this week, we’ve been able to play a baseline profile video without any particular quirks anymore. Some time was thus spent on cleaning up and refactoring the driver, libva-dump and cedrus-frame-test tools in order to support both the MPEG2 and H264 codecs, a feature that was dropped due to many hacks during the development. We then took the occasion to start the discussion on the linux-media mailing list by sending a preliminary version of the patches. We then worked on the real libva-cedrus, adding the support for H264. Most of the code is there now, but unfortunately isn’t functional yet. Some debugging will be on the agenda next week 🙂

On the players integration side, the goals for this week covered Kodi support for our beloved Allwinner platforms (of course, with upstream). But first, a few words as a follow-up to last week’s work on the MB32 untiling GPU shader. A specific commit related to texture uploading on the Mali400 was spotted in the MER project, fixing an issue apparently very similar to our own. Alas! It didn’t help with our case and did not lead to any improvement.

While the shader untiler is required for accelerated X11 display with the GPU, Kodi offers direct DRM/KMS support (the Kernel Mode-Setting part of the Display Rendering Manager, that deals with on-screen display). This means that we can use the DRM work from months ago for untiling the VPU buffers directly with the video engine. This is sometimes even faster than the GPU, especially for 4K contents!

However, Kodi is a complex piece of software that requires significant integration. Its support in Buildroot definitely reveals that complexity, that is gracefully abstracted by the build system. On top of that, the Kodi target platform for using DRM/KMS, called GBM (we’ll get back to this acronym in a bit) is not supported in most build systems (Buildroot included), with the exception of LibreELEC, that is used by the developers contributing to this Kodi target. After an intense struggle, it became clear that LibreELEC was the only reasonable and sane way to go for supporting GBM. Thanks to the huge help and incredible availability of the community of LibreELEC developers interested in Allwinner support, it was possible to finally bootstrap a working installation (that does not interface with our VAAPI backend yet):

In order to provide high performance and a pleasant experience, Kodi heavily relies on the GPU, which is supported by the EGL and GLES interfaces. EGL, in charge of the display part, has to be connected to the native windowing system of the target in use, that can be X11 or Wayland/GBM. GBM, which stands for Generic Buffer Management is an abstracted API for graphics-related memory management. It allows abstracting memory allocators such as GEM (the Graphics Execution Manager used in conjunction with DRM) through a consistent and unified interface that is, as for EGL and GLES, independent from the system and hardware implementations. Kodi uses GBM directly to allocate buffers shared between the GPU and the DRM subsystem.

This requires explicit cooperation from the used EGL implementation, the Mali blobs in our case. Sadly, the blobs available for the A10/A13/A20 and A33 platform do not provide the GBM interface. Still, LibreELEC offers support for the H3 platform and so it was selected as a primary target for setting up Kodi support for the GBM target. Thanks to Libre Computer, we received a (significant) number of boards for our development purposes, including H3 boards that were directly useful in this effort!

The H264 effort has also seen some great progress this week. We finally got the first frame of Big Buck Bunny to be decoded on Monday, and gradually improved the libva-dump, cedrus-frame-test and our kernel driver to fix the bugs that were found along the way. The libvdpau-sunxi authors, and Jens Kuske in particular, provided some great feedback on how the reference list, decoded buffer buffers, and the Video Engine in general were working. We now can play a video with only I-frames without any hiccups (that we found out), and the P-frame support is slowly getting into shape. We can decode the first 4 frames of Big Buck Bunny without any issue, and the fifth is reported as decoded, but, well, see below for yourself… Obviously this will need a bit more work, and to test it with other videos and with B-Frames. But this is good news!

Integration with video players

The work conducted this week on the video output side was focused on writing a shader for untiling the MB32 NV12-based format used by the VPU to output frames. This brought various challenges, some of which are presented below.

Since GLES and EGL are generic APIs that are not tied to a particular driver implementation, it made sense to start writing the shader on an x86 Intel-based device with GPU support in Mesa 3D (and speed-up the development time). The first step to the process was to display the raw pixel values from the luminance plane through the shader. Actually, two shaders are required: one for the vertex processor and one for the fragment (pixel) processor of the GPU. The former is in charge of applying geometrical operations to the vertices (the points that define the 3D mesh) while the latter defines the color for each rendered pixel from that mesh. In our case, the mesh is simply a rectangle that matches our window size. The tiled NV12 luminance plane is uploaded to the GPU as a 1-byte-per-pixel texture, which allows addressing each component separately. However, the coordinates for the texture are normalized by the GPU, so coordinates to retrieve texels (texture pixels) form the texture sampler are specified as decimal values. This makes it tricky to ensure that the right value is retrieved, especially given that the GPU might apply various filtering techniques (that are a really good thing to have when dealing with actual textures for 3D models, though).

Setting up the vertex and fragment shaders to linearly display the pixels from the tiled format results in a mangled display (as expected):

Thanks to the documentation made available by the linux-sunxi community, it was possible to rapidly draft a formula for getting the right texel location, that produced mitigated results:

With some extra work (and quirks for ensuring that the right texel is picked on tile edges), the luminance component was finally displayed correctly:

Next up was the chrominance component, that required importing a second dedicated texture. First tries lead to funky-looking coloring of the frame:

Until the shader was corrected to end up with a good-looking picture:

Real trouble began when porting this work to the Mali, that does not behave the same when it comes to texture uploads (and requires line-by-line upload for 1-byte-per-pixel formats). Since we are aiming at DMAbuf import instead of (slow) texture upload, no time was spent coping with the difference. The main issue with DMAbuf import is that the usual one-byte-per-pixel format (described by the DRM_FORMAT_R8 fourcc code) is simply not supported by the GPU, leaving only RGB and YUV as options, that do not directly fit the bill. We are still investigating ways to make our texture available to the GPU’s texture sampler without extra copies (or with copies that can fit our bill in terms of performance).

H264 support

We also worked on the H264 decoding in the kernel, and some progress was made. The libva-dump and cedrus-frame-test ports are now done, and we’ve been able to run cedrus-frame-test on 32 frames without any hiccups… Unfortunately, while the VPU reports the frames as properly decoded, the contents of the output buffer is blank, which is obviously not great. Since then, we have simplified the test to have a single frame decoded, and compared the register write sequence between libvdpau-sunxi and our kernel code. This has allowed us to find some bugs in the driver, but the current state is still that we can’t decode a frame. We shouldn’t be very far now though, so stay tuned for our next status update!

This week’s effort was focused on getting VLC to accelerate its video output using the Mali proprietary blobs. More specifically, two distinct interfaces are involved: EGL, that allows interacting with the platform’s windowing system (in our case, X11) and GLES, that is in charge of the rendering operations. While VLC already had support for both of these interfaces, it initially failed to create and use its GL-backed video output module with the Mali GPU blobs. Although everything indicated that it should have been working, the GLES calls were failing while EGL was setup and behaving correctly. The issue at hand was directly related to VLC using Qt for its interface. Because the Qt build used on the development boards was targeting GL support instead of GLES, it needed to import GL symbols that have the same name as GLES equivalents. Since Qt was loaded after the video output module, it would override the matching GLES symbols with GL symbols (from Mesa, not the blob).

With the help of Thomas Guillem, a few patches were crafted to fix the issue and sent out to the VLC developers. Some more revisions of these patches will be needed for the fix to integrate the VLC tree, but it should land sooner or later.

With VLC fixed, it was time to start looking at accelerating our pipeline with the GPU. VLC already includes GPU shaders for NV12 to RGB conversion as well as scaling and rotation, but does not have support for our tiled format. This is why we need a shader on our VAAPI backend side to accelerate the untiling operation. While the shader is currently work in progress, further work is also required to properly export the resulting untiled buffer as a DMABUF handle for VLC. Since the GPU blob does not support dmabuf export, we will need to implement a standalone GBM provider compatible with our DRM driver, that will handle allocating surfaces (instead of the armsoc DDX that is currently used for accelerated graphics on X) and exporting DMABUF handles to them when needed. Generally speaking, this will also allow standardizing and sanitizing the integration of the Mali blobs with the rest of the system.

Stay tuned for our next update!

With DMABUF support tested, it has become possible for Paul to start the work on integrating a GPU-based video output pipeline with Sunxi-Cedrus. Using the GPU should greatly improve performance when it comes to displaying the video frames. As of now, we have been using software untiling, software YUV-to-RGB colorspace conversion and software scaling. We are looking to replace these steps with GPU-based untiling, colorspace conversion and scaling. Shaders are used to implement these operations: they are small programs that are compiled on-the-fly for the GPU’s very specialized instruction set. Most players embed shaders (in their source form, using the GL shading language) for usual operations like colorspace conversion and scaling. However, these players are not ready to handle untiling as of now (or even be notified that the format returned by our VAAPI backend is tiled).

The first step in our plan is to get VLC to cooperate with the X11 flavor of the Mali proprietary blobs that Allwinner has released in the past so that we can use GPU support for colorspace and scaling. This is still a blocking point as of now. Then, we will look into crafting a shader for untiling the VPU output frame and integrating it with our libva-cedrus VAAPI backend.

As a sidenote, the free software Lima driver is being prepared for a first RFC series, bringing the first bits of mainline Linux kernel support for Mali GPUs of the Utgard generation. So even though work on the GPU only concerns the proprietary blob for now, the work will eventually become useful to the free software driver as well.

We have also tested Sunxi-Cedrus on the H3 and started looking at integrating the display part (which differs from earlier SoCs by using a revised display engine: DE2). However, since this is a strech goal of the fundraiser and that we have many other tasks left to tack among our main goals, this is by far not our priority at the moment.

We finally worked more on the libva-dump and cedrus-frame-test for H264, which will hopefully allow us to test our first H264 decoding next week!

Stay tuned for our next progress update!

This week has seen considerably less advancement than the ones before it due to bank holidays in France. Nevertheless, we managed to prepare and send V3 of the Sunxi-Cedrus Linux kernel driver on Monday. While this new version contains several incremental improvements, a number of tasks (described in the series’ cover letter) have yet to be completed before the driver can be merged in mainline Linux.

Maxime continued to work on the H264 support. The big part of the kernel has been done, and he then moved on to convert libva-dump to be able to dump also H264 buffers. Most of that part has been done as well, so the next item will be to convert cedrus-frame-test to be able to test H264 frames, and see where that takes us.

Paul kept working on DMABUF support, which is now refined and ready both on the kernel side and on the userspace side with cedrus-frame-test. There is now a single DMABUF handle used per buffer plane (instead of per-plane) which allows having all components of the frame displayed correctly. Because there is now as many buffers for display as there are for decoding, it is necessary to register framebuffers associated with each imported buffer and cycle the framebuffers in multi-buffering page-flipping. To tackle this, we have started implementing atomic modesetting in cedrus-frame-test, allowing to set the framebuffer to use per-plane.

Finally, some attention was given to the integration of our video decoding pipeline with the Mali GPU, especially to target Kodi support.

Stay tuned for our next update!