Initial Allwinner V3 ISP support in mainline Linux

Introduction

Several months ago, Bootlin announced ongoing work on MIPI CSI-2 support for the Allwinner A31/V3 and A83T platforms in mainline Linux, as well as support for the Omnivision OV8865 and OV5648 image sensors. This effort has been a success and while the sensor patches were already integrated in mainline Linux since, the MIPI CSI-2 controller patches are on their way towards inclusion.

Since then we had the opportunity to take things further and start tackling the next steps for advanced camera support in mainline Linux on Allwinner SoCs!

With MIPI CSI-2 support and proper sensor drivers available in V4L2, we were able to capture raw bayer data provided by the sensors. But this data does not constitute the final picture that can be displayed or encoded into a file: a number of enhancement and transformation steps are required to achieve a visually-pleasing result that users typically expect.

These steps are quite calculation-intensive and it does not make sense to implement them with a software pipeline, especially with rates of 25, 30 or even 60 frames per second that are typically expected for video recording.

An open-source and upstream driver for the Allwinner ISP

Allwinner SoCs that support MIPI CSI-2 also include an Image Signal Processor hardware unit, a dedicated accelerator for enhancing and transforming raw data received from sensors.

Allwinner ISP features
Features of the ISP as described in the Allwinner V3 datasheet.

Since support for this ISP was implemented using a non-free blob in Allwinner SDKs, this area remained unsupported in mainline Linux… Until now!

Thanks to some help from Allwinner, we were able to implement a proper V4L2 driver for the Allwinner ISP found in the Allwinner V3, completely open-source, with no binary blob involved. This work was recently submitted upstream, with a first revision totaling more than 8000 new lines of code, which comes together with a significant rework of the Allwinner camera interface driver to make it usable with or without the ISP, and including the new MIPI CSI-2 support which we had submitted previously. We are very happy to keep contributing to advancing fully open-source Allwinner SoCs support in mainline Linux and help tackle some of the remaining areas there!

Our currently proposed driver for the Allwinner ISP only supports a limited set of features: debayering with coefficients and 2D noise filtering. These features were sufficient for our use case, and allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator.

As the driver for now relies on a specific user-space API that does not yet cover all aspects of the ISP, the driver was submitted to the Linux kernel staging area and will probably stay there until all ISP features are properly described.

Our work on this advanced camera support, including the ISP driver, has been described in the talk we have given earlier this week at the Embedded Linux Conference, for which the slides are already available.

Advanced camera support on Allwinner SoCs with Mainline Linux

Additional features and future work

The Allwinner ISP supports a lot more features beyond just debayering and noise filtering. For example, it supports statistics to implement 3A algorithms (auto-focus, auto-exposition and auto-white-balance) which are necessary to avoid manual configuration of scene-specific parameters. These could typically be implemented in libcamera, the community free software project that supports complex image capture pipelines and ISPs.

As a result Bootlin would be very interested to continue this work and bring this driver to a more advanced state. So if you have a project that could help move this topic forward, do not hesitate to contact us about it!

Author: Paul Kocialkowski

Paul is a kernel and embedded Linux engineer at Bootlin, which he joined in 2018. See More details...

30 thoughts on “Initial Allwinner V3 ISP support in mainline Linux”

    1. You can do reverse engineering by observing the registers written by a closed-source binary. In this specific case, as mentioned in the blog post, we’ve got some help from Allwinner. We can’t really go much into the details about it in this specific case. We didn’t receive a datasheet, but enough information to help us write the ISP driver.

      Reply

  2. Hello, I have some questions.
    How about support Video Encoder? will it be released anytime soon?
    At the moment I’m using to encode video this library: https://github.com/aodzip/cedar with ffmpeg+omx plugin and can encode 640×480@50fps max, I get 100% cpu load. Using ftrace I found what the reason related to something v4l2 copy_to_user.
    Can it related with debayering process? I see it is written “allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator”.
    Can I get low cpu load if I use mainline ISP?

    Reply

    1. Supporting the video encoder would be great and Bootlin is definitely interested in working in that direction, but at this point the work has not started.
      The ISP is not related to the encoder so I don’t think that it will help with your CPU load issue.

      Reply

    3. Which command do you use for capturing and encoding the video? I use the same library and can only achieve realtime encoding with 320×280 resolution. With 800×600 I can get 8 FPS.
      This makes the encoder nearly useless for my project. I hope you can help me.

      Reply

      1. I work with my own code, which is available at https://github.com/paulkocialkowski/v4l2-bayer
        otherwise I also use fswebcam for standalone capture.

        Generally speaking I only work with single shots at this point, recording is not yes a possibility and using a software encoder might be pretty slow. Also note that the memory allocated with v4l2 is not cached, which makes cpu access pretty slow too.

        Reply

  3. I’m trying to build this driver in linux-5.13, I get many errors in sun6i_isp_proc.c file.
    This one of this:
    drivers/staging/media/sunxi/sun6i-isp/sun6i_isp_proc.c:250:38: error: passing argument 2 of ‘v4l2_subdev_get_try_format’ from incompatible pointer type [-Werror=incompatible-pointer-types]
    250 | v4l2_subdev_get_try_format(subdev, state, pad);
    | ^~~~~
    | |
    | struct v4l2_subdev_state *
    In file included from ./include/media/v4l2-device.h:13,
    from drivers/staging/media/sunxi/sun6i-isp/sun6i_isp_proc.c:9:

    I applied all patches. Which kernel version is compatible with this ISP driver?

    Reply

    1. Hi Niyaz,

      The driver is based on a recent media tree, not the v5.13 release. The base commit I used is: 50e7a31d30e8221632675abed3be306382324ca2

      I hope this will be helpful to you!

      Reply

  4. hi Mr. Paul Kocialkowski
    thanks for your great work
    based on this section:
    “Our currently proposed driver for the Allwinner ISP only supports a limited set of features: debayering with coefficients and 2D noise filtering. These features were sufficient for our use case, and allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator.”

    have you compare the result with another camera(such as cctv camera with sony’s sernsor) with 5 megapixel sensor?

    Reply

    1. I didn’t do any comparison of the debayering results but IIRC Allwinner docs mention that it’s more advanced than linear interpolation so maybe something like VPG (which seems to be very common).

      Reply

  5. I have IMX219 camera and lichee pi zero board.
    Could you please give me dts description for trying ISP driver debayering for imx219? I cannot find on the Internet any example of dts for imx219 with v3s

    Reply

    1. Hi,

      You can find a description for the device-tree in the bindings, at: https://lore.kernel.org/linux-media/20210910184147.336618-18-paul.kocialkowski@bootlin.com/

      Generally speaking, you still need to attach your sensor to the CSI node and make sure you enable the ISP driver. The drivers should then be binded together (check the topology with media-ctl -p).

      Feel free to check the imx219 bindings documentation to integrate it depending on your setup (especially clocks and power supplies).

      Reply

      2. I got below message could you please help what shoud do?

        v4l2-ctl –device /dev/video0 –stream-mmap –stream-to=AAA.raw –stream-coun
        t=1

        [ 277.264405] cma: cma_alloc: reserved: alloc failed, req-size: 1013 pages, ret: -12
        [ 277.272206] sun6i-csi 1cb0000.camera: dma alloc of size 4149248 failed
        VIDIOC_STREAMON returned -1 (Link has been severed)

        Reply

  6. Paul, Are you working on other ISP functions? will it be released?
    ISP driver is applicable to allwinner S3?

    Reply

    1. No, we are not working on other ISP functions, as the debayering and noise filtering were the only two features needed by our customer. We can develop extra features if there is interest from other customers.

      Paul has recently posted a new iteration of a large page series that includes the ISP driver, see https://lore.kernel.org/linux-media/20220205185429.2278860-1-paul.kocialkowski@bootlin.com/.

      Regarding the applicability of the driver to Allwinner S3, I will let Paul comment on this. Though based on typical Allwinner behavior, I would assume that there is a very high chance for the S3 to use the same ISP.

      Reply

      1. Hello Paul and Thomas,

        I’m a little confused about the difference between of these two sets of patches:

        1. https://lore.kernel.org/linux-media/20210910184147.336618-1-paul.kocialkowski@bootlin.com/
        2. https://lore.kernel.org/linux-media/20220205185429.2278860-1-paul.kocialkowski@bootlin.com/

        The first was posted in September 2021 and has 22 items, while the second was posted in February 2022 and has 66 items. Why does the 2nd one have significantly more items? Which one should I use to experiment with MIPI CSI-2 and the ISP?

        Thank you.

        Reply

        1. The second iteration splits the sun6i-csi rework into different commits instead of removing the current driver and adding a new one. Both of these iterations are now deprecated. The current series is split into 4 series (platform, mipi csi-2, csi rework, isp), with the mipi csi-2 series already merged for 6.0.

          You can find the latest versions of the series on the linux media patchwork: https://patchwork.linuxtv.org/project/linux-media/list/?submitter=7413

          Reply

          1. Got it, thank you so much! By the way, is there a userspace demo program available somewhere? I’m having some trouble understanding how to get the ISP to work.

            Reply

        1. It was based on the linux media tree at the time, but these series are now outdated.
          Generally speaking, patches sent on the media mailing list are supposed to apply on top of the latest media tree, which is itself based on (more or less) the latest rc from Linus.

          Reply

  7. Hi,
    I do not understand what is the obstacle of being a close source isp? A software for running on a PC and an instruction provided by allwinner according to which that blob file containing isp parameters can be generated for a specific sensor. What is the problem now? Is there an obstacle in using that generated file and preparing a driver for our new sensor?
    I probably do not know anything. Thank you for guiding me
    nimadibaj@seznam.cz

    Reply

  8. Hi Paul,

    Could you please help, still can not get the video to work.
    Is there any missing for me?
    below the test result:

    v4l2-compliance
    v4l2-compliance 1.22.1, 32 bits, 32-bit time_t

    Compliance test for sun6i-csi device /dev/video0:

    Driver Info:
    Driver name : sun6i-csi
    Card type : sun6i-csi-capture
    Bus info : platform:1cb0000.camera
    Driver version : 6.2.0
    Capabilities : 0x84200001
    Video Capture
    Streaming
    Extended Pix Format
    Device Capabilities
    Device Caps : 0x04200001
    Video Capture
    Streaming
    Extended Pix Format
    Media Driver Info:
    Driver name : sun6i-isp
    Model : Allwinner A31 ISP Device
    Serial :
    Bus info : platform:1cb8000.isp
    Media version : 6.2.0
    Hardware revision: 0x00000000 (0)
    Driver version : 6.2.0
    Interface Info:
    ID : 0x0300000f
    Type : V4L Video
    Entity Info:
    ID : 0x0000000d (13)
    Name : sun6i-csi-capture
    Function : V4L2 I/O
    Pad 0x0100000e : 0: Sink, Must Connect
    Link 0x02000011: from remote pad 0x1000007 of entity ‘sun6i-csi-bridge’ (Video Interface Bridge): Data

    Required ioctls:
    test MC information (see ‘Media Driver Info’ above): OK
    warn: v4l2-compliance.cpp(642): media bus_info ‘platform:1cb8000.isp’ differs from V4L2 bus_info ‘platform:1cb0000.camera’
    test VIDIOC_QUERYCAP: OK
    test invalid ioctls: OK

    Allow for multiple opens:
    test second /dev/video0 open: OK
    warn: v4l2-compliance.cpp(642): media bus_info ‘platform:1cb8000.isp’ differs from V4L2 bus_info ‘platform:1cb0000.camera’
    test VIDIOC_QUERYCAP: OK
    test VIDIOC_G/S_PRIORITY: OK
    test for unlimited opens: OK

    Debug ioctls:
    test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
    test VIDIOC_LOG_STATUS: OK (Not Supported)

    Input ioctls:
    test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
    test VIDIOC_ENUMAUDIO: OK (Not Supported)
    test VIDIOC_G/S/ENUMINPUT: OK
    test VIDIOC_G/S_AUDIO: OK (Not Supported)
    Inputs: 1 Audio Inputs: 0 Tuners: 0

    Output ioctls:
    test VIDIOC_G/S_MODULATOR: OK (Not Supported)
    test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
    test VIDIOC_ENUMAUDOUT: OK (Not Supported)
    test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
    test VIDIOC_G/S_AUDOUT: OK (Not Supported)
    Outputs: 0 Audio Outputs: 0 Modulators: 0

    Input/Output configuration ioctls:
    test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
    test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
    test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
    test VIDIOC_G/S_EDID: OK (Not Supported)

    Control ioctls (Input 0):
    test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
    test VIDIOC_QUERYCTRL: OK (Not Supported)
    test VIDIOC_G/S_CTRL: OK (Not Supported)
    test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
    test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
    test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
    Standard Controls: 0 Private Controls: 0

    Format ioctls (Input 0):
    fail: v4l2-test-formats.cpp(263): fmtdesc.description mismatch: was ‘Y/UV 4:2:2’, expected ‘Y/CbCr 4:2:2’
    test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: FAIL
    test VIDIOC_G/S_PARM: OK (Not Supported)
    test VIDIOC_G_FBUF: OK (Not Supported)
    test VIDIOC_G_FMT: OK
    test VIDIOC_TRY_FMT: OK
    test VIDIOC_S_FMT: OK
    test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
    test Cropping: OK (Not Supported)
    test Composing: OK (Not Supported)
    test Scaling: OK

    Codec ioctls (Input 0):
    test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
    test VIDIOC_G_ENC_INDEX: OK (Not Supported)
    test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

    Buffer ioctls (Input 0):
    test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
    test VIDIOC_EXPBUF: OK
    test Requests: OK (Not Supported)
    test TIME32/64: OK

    Total for sun6i-csi device /dev/video0: 47, Succeeded: 46, Failed: 1, Warnings: 2

    Reply

Leave a Reply