Back in early 2018, Bootlin started a crowd-funding campaign to fund the development of an upstream Linux kernel driver for the VPU found in Allwinner processors. Thanks to the support from over 400 contributors, companies and individuals, we have been able to bring support for hardware-accelerated video decoding in the mainline Linux kernel for Allwinner platforms.
From April 2018 to end of 2019, Paul Kocialkowski and Maxime Ripard at Bootlin worked hard on developing the driver and getting it accepted upstream, as well as developing the corresponding user-space components. We regularly published the progress of our work on this blog.
As of the end of 2019, we can say that all the goals defined in the Kickstarter have been completed:
- We have an upstream Linux kernel for the Allwinner VPU, in drivers/staging/media/sunxi/cedrus, which supports MPEG2 decoding (since Linux 4.19), H264 decoding (since Linux 5.2) and H265 decoding (will be in the upcoming Linux 5.5)
- We have a user-space VA-API implementation called libva-v4l2-request, and which allows to use any Linux kernel video codec based on the request API.
- We have enabled the Linux kernel driver on all platforms we listed in our Kickstarter campaign: A13/A10S/A20/A33/H3 (since Linux 4.19), A64/H5 (since Linux 4.20), A10 (since Linux 5.0) and H6 (since Linux 5.1, contributed by Jernej Skrabec)
This means that the effort that was funded by the Kickstarter campaign is now over, and from now on, we are operating in maintenance mode regarding the cedrus driver: we are currently not actively working on developing new features for the driver anymore.
Of course, there are plenty of additional features that can be added to the driver: support for H264 encoding, support for high-profile H264 decoding, support for other video codecs. Bootlin is obviously available to develop those additional features for customers, do not hesitate to contact us if you are interested.
Overall, we found this experience of funding upstream Linux kernel development through crowd-funding very interesting and we’re happy to have been successful at delivering what was promised in our campaign. Looking at the bigger picture, the Linux userspace API for video decoding with stateless hardware codecs in V4L2 has been maturing for a while and is getting closer and closer to being finalized and declared a stable kernel API: this project has been key in the introduction of this API, as cedrus was the first driver merged to require and use it. Additional drivers are appearing for other stateless decoding engines, such as the Hantro G1 (found in Rockchip, i.MX and Microchip platforms) or the rkvdec engine. We are of course also interested in working on support for these VPUs, as we have gained significant familiarity with all things related to hardware video decoding during the cedrus adventure.
I was glad to contribute to the funding for this and would gladly do so in the future if you choose to use that model again. I think it’s safe to say that you met all of your obligations and went above and beyond as well.
Thank you and all of your staff who put their time and effort into this.
Thanks a lot for your nice comment! We will indeed be thinking of using a similar model for other developments in the future. Stay tuned!
As part of this funding I only can say million thanks, grand merci, un millón de gracias.
And count with me in future upstream Linux kernel development.
It is a totally different area and Bootlin may not be interested in that, but maybe your crowdfunding experience can boost the libinput project.
Read also: https://who-t.blogspot.com/2019/10/libinputs-bus-factor-is-1.html
I funded the project but I’m not completely convinced… but now guys you can count with me forever. Glad to support open source projects.
Hello Manuti. Thanks for your comment. Could you explain why you are not completely convinced? We believe we delivered all the goals we had defined in our Kickstarter campaign, and practically speaking there are now Librelec images that use the work we have done on the Allwinner VPU kernel driver, so it is really being used in real life. We will be happy to give more details if there is anything unclear in what we delivered.
I guess he was sceptical at the beginning, but now hes convinced.
Honestly I was also very much ambiguous when you started the campaign, so I didn’t donate, just advertised for it wherever I could. In the meantime I saw what you did, what’s the outcome of your efforts and it made my useless tv box a great libreelec player. Jernej really made your efforts generally accessible. In any case it is in a way a shame you had to get money via crowdfunding and didn’t get it from allwinner. Let’s hope this will convince some other semiconductor companies opening up to mainline Linux, especially if after LE now also pinephone will take off and replicant will hopefully prove how far it can go with android if linux philosophy is applied instead of blob “kanging” as usual for android. Thanks for all your efforts, not only the ones for the vpu.
Thanks for your comment. Glad to hear that we have been able to prove you that we could do it, and that we fulfilled the goals set by our Kickstarter campaign.
Thanks. I look forward to hearing about similar projects in the future. Maybe for example making more software take advantage of VA-API.
What’s the plan with maintaining the libva-v4l2-request library?
Hello plaes, thanks for your question. We will continue to maintain the libva-v4l2-request library: Paul is still given some time for maintenance. What we will not do unless we have some customers asking for it is implementing new features, such as encoding support or H264 high-profile support: the effort is quite significant.
Are there plans to tag a new release of libva-v4l2-request with H.265 support?
That would be a good idea, yes! H.265 support in the library was merged in the master branch a while ago, but we have not made a new release of it or updated instructions in particular yet. Now that the code has landed upstream, it would be a good time to do so.
Which H264 high-profile features are missing? AFAIK only interlace content decoding doesn’t work yet.
I still feel like lacking visibility on exactly which features of high-profile H.264 are working and more generally, what the overall test coverage is. When H.264 supported was merged, I believe there were missing features. Indeed, since there were contributions (mostly by you) in the meantime, the situation must have improved to a large extent (thanks for that)! There are also possibly features that are not well supported by our uAPI (e.g. multi-layers, multi-views) that still need some investigation.
Great news! Any tutorial on how to utilize VE with recent drivers?
We prepared a wiki page at https://linux-sunxi.org/Sunxi-Cedrus that provides the bulk of instructions and releases. However, it is not always straightforward to find the corresponding version of each component to make them work together. We’ll try to improve this and update the instructions in the future.
Would it be possible for you guys to give that wiki page an overhaul, now that the upstreaming is done?
Thank you for proving that crowdfunding can be used to develop drivers for the upstream kernel.
The development of the Pinephone might offer the opportunity to start another Kickstarter. The drivers for the A64 used by the Pinephone seam to have some limitations like the lack of rotation in the video engine which is a pity for a phone. Also, this platform seems to have gotten people interested.
Does Bootlin have the ability to gauge interest with the backers of the completed Kickstarter?
Hello Xaviour. Thanks for your comment. We are also happy to have proven that indeed upstream kernel development can be done through crowdfunding.
We will evaluate and discuss what are potential topics for other crowd-funding campaigns, but in general we will probably want to target them at solving fairly complex/risky topics, which are too complex/risky for a single small company to fund. The VPU support was in this category: it was risky, and required lots of work with the upstream kernel community to figure the right API for stateless video codecs. On the other hand, something like display rotation is normally simpler, and could easily be funded by a company, even small.
Maybe the example given was chosen poorly and confused the main issue. The underlying idea was to point out the fact that there is an opportunity. For the first time in many years, there is a device that is affordable and powerful enough to be useful to a large enough number of people which means that there might be enough people willing to finance to improve its hardware support. Reaching it is also simplified because the community is quite organized. The company that makes the platform takes over the risk of hardware development and selling a device with a small margin but leaves the risk of software development to the community. Finally, it just happens to be based on a platform that Bootlin has experience with.
Concretely, in order for the platform to be useful as a daily driver, its power consumption must be low enough otherwise it will only be used as a cheap development platform. While there is a community based effort to write drivers and firmware for the power management chip, the work is not done and has not guaranty to be complete in a timely manner. Another big contributor to power consumption is the 4G radio. The current solution might be good enough for a laptop with a 3G/4G card or USB key but is it good enough for a smartphone with a much smaller battery?
It has been a great work. Hope in the future AV1 codec could be implemented!
As far as we know, Allwinner SoCs currently do not include AV1 hardware decoding. If that becomes the case, we’ll definitely keep an eye on the topic!
Hi!
At first I’d like to thank you for the good work on this and all your other open source projects.
I wonder if you have any information on how to use the driver on a Pocket CHIP as the one you used on a photo in the Kickstarter campaign.
The Allwinner VPU support has nothing board-specific, so the instructions provided at https://linux-sunxi.org/Sunxi-Cedrus will work for the PocketCHIP as well, as the PocketCHIP runs an Allwinner A13 processor. Of course, how easy it will be to apply these instructions to your particular system depends on what is the operating system you run on your PocketCHIP, which version it is, how you build it, etc.
Thanks! I’ll read my way through it 😊
First, thanks a lot for your efforts (and specially for going the extra mile). I’ve donated although I don’t own any Allwinner gear (and don’t plan to), just to get more open options later on (and there is karma: the soc I use has now received a great driver funded by somebody else).
After this success, what about crowdfunding UHS II and III support? At least some of you know this topic very well from the start! (http://events17.linuxfoundation.org/sites/events/files/slides/clement-sd-mmc-high-speed-support-in-linux-kernel.pdf)…
Hi. Thanks for your effort. Quick question. Is it possible to use GStreamer for H.265 decoding already?
Hi, how to use Cedrus on A23? sun8i-a23.dtsi has no video-codec inside, and if i try to copy it from sun8i-a33.dtsi it’s not working.
It’s pretty likely that the A23 has the same VPU, so it should be mostly a matter of adding the right description in sun8i-a23.dtsi, but it indeed is probably not just a copy/paste from sun8i-a33.dtsi. Please contact us for commercial support on Allwinner/Linux support if you’re interested, otherwise you can also reach out to the linux-sunxi community (which we are also part of).
Thanks for the hard work! What’s the easiest way to use these drivers at the moment with gstreamer for h618?
We don’t yet have H616/H618 support for cedrus in mainline, although it’s probably very much doable to add it.
In GStreamer you should use the v4l2codecs implementation with the
v4l2sl*decelements such as:$ gst-launch-1.0 filesrc location=samples/caminandes-gran-dillama-720p-vp8.webm ! parsebin ! v4l2slvp8dec ! kmssink plane_id=31 connector_id=51 plane-properties=s,zpos=2This would pipe the output to a kms plane directly and set its z-pos to be atop the primary plane.
Hello, I would like to ask if there is any plan to upgrade to mainline kernel for lipva-v4l2-request. As a novice, I want to use it in H618, but it is very sad that I cannot use it
Hello, is there any plan to update to mainline kernel version for lipva-v4l2-request, newbies would like to use it but can’t