Tag: cedrus

Wrapping up the Allwinner VPU crowdfunded Linux driver work

Back in early 2018, Bootlin started a crowd-funding campaign to fund the development of an upstream Linux kernel driver for the VPU found in Allwinner processors. Thanks to the support from over 400 contributors, companies and individuals, we have been able to bring support for hardware-accelerated video decoding in the mainline Linux kernel for Allwinner platforms.

From April 2018 to end of 2019, Paul Kocialkowski and Maxime Ripard at Bootlin worked hard on developing the driver and getting it accepted upstream, as well as developing the corresponding user-space components. We regularly published the progress of our work on this blog.

As of the end of 2019, we can say that all the goals defined in the Kickstarter have been completed:

  • We have an upstream Linux kernel for the Allwinner VPU, in drivers/staging/media/sunxi/cedrus, which supports MPEG2 decoding (since Linux 4.19), H264 decoding (since Linux 5.2) and H265 decoding (will be in the upcoming Linux 5.5)
  • We have a user-space VA-API implementation called libva-v4l2-request, and which allows to use any Linux kernel video codec based on the request API.
  • We have enabled the Linux kernel driver on all platforms we listed in our Kickstarter campaign: A13/A10S/A20/A33/H3 (since Linux 4.19), A64/H5 (since Linux 4.20), A10 (since Linux 5.0) and H6 (since Linux 5.1, contributed by Jernej Skrabec)

This means that the effort that was funded by the Kickstarter campaign is now over, and from now on, we are operating in maintenance mode regarding the cedrus driver: we are currently not actively working on developing new features for the driver anymore.

The Cedrus Linux driver in action

Of course, there are plenty of additional features that can be added to the driver: support for H264 encoding, support for high-profile H264 decoding, support for other video codecs. Bootlin is obviously available to develop those additional features for customers, do not hesitate to contact us if you are interested.

Overall, we found this experience of funding upstream Linux kernel development through crowd-funding very interesting and we’re happy to have been successful at delivering what was promised in our campaign. Looking at the bigger picture, the Linux userspace API for video decoding with stateless hardware codecs in V4L2 has been maturing for a while and is getting closer and closer to being finalized and declared a stable kernel API: this project has been key in the introduction of this API, as cedrus was the first driver merged to require and use it. Additional drivers are appearing for other stateless decoding engines, such as the Hantro G1 (found in Rockchip, i.MX and Microchip platforms) or the rkvdec engine. We are of course also interested in working on support for these VPUs, as we have gained significant familiarity with all things related to hardware video decoding during the cedrus adventure.

Bootlin at the Linux Media Summit 2018

This year’s edition of the Linux Media Summit happened a month ago, in Edinburgh, right after the Embedded Linux Conference. Since we were already at the ELCE, and that we’ve been more and more involved in the media community thanks to our work on the Allwinner CSI driver and more importantly the Cedrus driver, it was natural for us to attend.

The media summit is usually a meeting to discuss the hot topics, so the whole day was a mix and match of various status updates and discussions on the future needs and developments around the Video4Linux2 framework.

Linux Media Summit attendees

Most of the discussion was about how to improve the contributor’s experience and improve the maintenance. The DRM subsystem was used as an example, since the number of patches are in the same order of magnitude, and a number of v4l2 contributors are also contributing to DRM drivers. Part of the improvement of both the maintenance and contribution experience will also come through some CI work, so there was a lot of discussions on how to improve the already existing tools (such as v4l2-compliance) but also how to setup some automatic tooling to run those tests as early as possible.

A good part of the day was also spent on dealing with the current developments, such as the Request API we’ve used in the Cedrus driver, and how to integrate that API into popular multimedia frameworks like gstreamer or ffmpeg. It looks like our libva implementation was well received, so it will probably be made standard and hosted on linuxtv.org in the near future. Other developments discussed were fault tolerant v4l2, in order to deal with video pipelines where one or several components might not work anymore, and storing the v4l2 controls state in a persistent way.

It was overall a very productive day, and it’s always nice to meet people you interact with over mailing list and IRC on a regular basis. If you want more information, you can read the extensive report.

Allwinner VPU support in mainline Linux November status update

Since our previous update back in September, we continued the work to reach the goals set by our crowdfunding campaign and made a number of steps forward. First, we are happy to announce that the core of the Cedrus driver was approved by the linux-media maintainers! It followed the final version of the media request API (the required piece of media framework plumbing necessary for our driver).

Both the API and our driver were merged in time for Linux 4.20, that is currently at the release candidate stage and will be released in a few weeks. The core of the Cedrus driver that is now in Linus’ tree supports hardware-accelerated video decoding for the MPEG-2 codec. We have even already seen contributions from the community, including minor fixes and improvements!

We have also been following-up on the other features covered by our crowdfunding campaign and made good progress on bringing them forward:

  • The series bringing H.264 decoding to our driver was updated for a second revision on November 15, rebased atop the upcoming Linux release and including a number of fixes as well as documentation;
  • H.265 decoding support followed with a second version sent on November 23, based on the updated H.264 series and bringing various minor improvements over the first iteration;
  • The patch series for the display engine DRM driver that adds support for the tiled YUV format used by the VPU was also updated, significantly reworked and submitted again on November 23;
  • Finally, we submitted a patch series adding support for the A64 and H5 Allwinner SoCs in the Cedrus VPU driver on November 15.
A64 and H5 boards provided by our sponsors Olimex and Libretech, now supported in Cedrus

With these patch series well on their way, we are closer than ever to delivering the remaining goals of the crowdfunding campaign!