Initial Allwinner V3 ISP support in mainline Linux

Introduction

Several months ago, Bootlin announced ongoing work on MIPI CSI-2 support for the Allwinner A31/V3 and A83T platforms in mainline Linux, as well as support for the Omnivision OV8865 and OV5648 image sensors. This effort has been a success and while the sensor patches were already integrated in mainline Linux since, the MIPI CSI-2 controller patches are on their way towards inclusion.

Since then we had the opportunity to take things further and start tackling the next steps for advanced camera support in mainline Linux on Allwinner SoCs!

With MIPI CSI-2 support and proper sensor drivers available in V4L2, we were able to capture raw bayer data provided by the sensors. But this data does not constitute the final picture that can be displayed or encoded into a file: a number of enhancement and transformation steps are required to achieve a visually-pleasing result that users typically expect.

These steps are quite calculation-intensive and it does not make sense to implement them with a software pipeline, especially with rates of 25, 30 or even 60 frames per second that are typically expected for video recording.

An open-source and upstream driver for the Allwinner ISP

Allwinner SoCs that support MIPI CSI-2 also include an Image Signal Processor hardware unit, a dedicated accelerator for enhancing and transforming raw data received from sensors.

Allwinner ISP features
Features of the ISP as described in the Allwinner V3 datasheet.

Since support for this ISP was implemented using a non-free blob in Allwinner SDKs, this area remained unsupported in mainline Linux… Until now!

Thanks to some help from Allwinner, we were able to implement a proper V4L2 driver for the Allwinner ISP found in the Allwinner V3, completely open-source, with no binary blob involved. This work was recently submitted upstream, with a first revision totaling more than 8000 new lines of code, which comes together with a significant rework of the Allwinner camera interface driver to make it usable with or without the ISP, and including the new MIPI CSI-2 support which we had submitted previously. We are very happy to keep contributing to advancing fully open-source Allwinner SoCs support in mainline Linux and help tackle some of the remaining areas there!

Our currently proposed driver for the Allwinner ISP only supports a limited set of features: debayering with coefficients and 2D noise filtering. These features were sufficient for our use case, and allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator.

As the driver for now relies on a specific user-space API that does not yet cover all aspects of the ISP, the driver was submitted to the Linux kernel staging area and will probably stay there until all ISP features are properly described.

Our work on this advanced camera support, including the ISP driver, has been described in the talk we have given earlier this week at the Embedded Linux Conference, for which the slides are already available.

Advanced camera support on Allwinner SoCs with Mainline Linux

Additional features and future work

The Allwinner ISP supports a lot more features beyond just debayering and noise filtering. For example, it supports statistics to implement 3A algorithms (auto-focus, auto-exposition and auto-white-balance) which are necessary to avoid manual configuration of scene-specific parameters. These could typically be implemented in libcamera, the community free software project that supports complex image capture pipelines and ISPs.

As a result Bootlin would be very interested to continue this work and bring this driver to a more advanced state. So if you have a project that could help move this topic forward, do not hesitate to contact us about it!

Author: Paul Kocialkowski

Paul is a kernel and embedded Linux engineer at Bootlin, which he joined in 2018. See More details...

10 thoughts on “Initial Allwinner V3 ISP support in mainline Linux”

    1. You can do reverse engineering by observing the registers written by a closed-source binary. In this specific case, as mentioned in the blog post, we’ve got some help from Allwinner. We can’t really go much into the details about it in this specific case. We didn’t receive a datasheet, but enough information to help us write the ISP driver.

  1. Hello, I have some questions.
    How about support Video Encoder? will it be released anytime soon?
    At the moment I’m using to encode video this library: https://github.com/aodzip/cedar with ffmpeg+omx plugin and can encode 640×480@50fps max, I get 100% cpu load. Using ftrace I found what the reason related to something v4l2 copy_to_user.
    Can it related with debayering process? I see it is written “allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator”.
    Can I get low cpu load if I use mainline ISP?

    1. Supporting the video encoder would be great and Bootlin is definitely interested in working in that direction, but at this point the work has not started.
      The ISP is not related to the encoder so I don’t think that it will help with your CPU load issue.

  2. I’m trying to build this driver in linux-5.13, I get many errors in sun6i_isp_proc.c file.
    This one of this:
    drivers/staging/media/sunxi/sun6i-isp/sun6i_isp_proc.c:250:38: error: passing argument 2 of ‘v4l2_subdev_get_try_format’ from incompatible pointer type [-Werror=incompatible-pointer-types]
    250 | v4l2_subdev_get_try_format(subdev, state, pad);
    | ^~~~~
    | |
    | struct v4l2_subdev_state *
    In file included from ./include/media/v4l2-device.h:13,
    from drivers/staging/media/sunxi/sun6i-isp/sun6i_isp_proc.c:9:

    I applied all patches. Which kernel version is compatible with this ISP driver?

    1. Hi Niyaz,

      The driver is based on a recent media tree, not the v5.13 release. The base commit I used is: 50e7a31d30e8221632675abed3be306382324ca2

      I hope this will be helpful to you!

  3. hi Mr. Paul Kocialkowski
    thanks for your great work
    based on this section:
    “Our currently proposed driver for the Allwinner ISP only supports a limited set of features: debayering with coefficients and 2D noise filtering. These features were sufficient for our use case, and allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator.”

    have you compare the result with another camera(such as cctv camera with sony’s sernsor) with 5 megapixel sensor?

    1. I didn’t do any comparison of the debayering results but IIRC Allwinner docs mention that it’s more advanced than linear interpolation so maybe something like VPG (which seems to be very common).

Leave a Reply to Paul Kocialkowski Cancel reply