Tag: isp

Introduction

Several months ago, Bootlin announced ongoing work on MIPI CSI-2 support for the Allwinner A31/V3 and A83T platforms in mainline Linux, as well as support for the Omnivision OV8865 and OV5648 image sensors. This effort has been a success and while the sensor patches were already integrated in mainline Linux since, the MIPI CSI-2 controller patches are on their way towards inclusion.

Since then we had the opportunity to take things further and start tackling the next steps for advanced camera support in mainline Linux on Allwinner SoCs!

With MIPI CSI-2 support and proper sensor drivers available in V4L2, we were able to capture raw bayer data provided by the sensors. But this data does not constitute the final picture that can be displayed or encoded into a file: a number of enhancement and transformation steps are required to achieve a visually-pleasing result that users typically expect.

These steps are quite calculation-intensive and it does not make sense to implement them with a software pipeline, especially with rates of 25, 30 or even 60 frames per second that are typically expected for video recording.

An open-source and upstream driver for the Allwinner ISP

Allwinner SoCs that support MIPI CSI-2 also include an Image Signal Processor hardware unit, a dedicated accelerator for enhancing and transforming raw data received from sensors.

Since support for this ISP was implemented using a non-free blob in Allwinner SDKs, this area remained unsupported in mainline Linux… Until now!

Thanks to some help from Allwinner, we were able to implement a proper V4L2 driver for the Allwinner ISP found in the Allwinner V3, completely open-source, with no binary blob involved. This work was recently submitted upstream, with a first revision totaling more than 8000 new lines of code, which comes together with a significant rework of the Allwinner camera interface driver to make it usable with or without the ISP, and including the new MIPI CSI-2 support which we had submitted previously. We are very happy to keep contributing to advancing fully open-source Allwinner SoCs support in mainline Linux and help tackle some of the remaining areas there!

• [PATCH 00/22] Allwinner A31/A83T MIPI CSI-2 Support and A31 ISP Support
  • [PATCH 01/22] clk: sunxi-ng: v3s: Make the ISP PLL clock public
  • [PATCH 02/22] ARM: dts: sun8i: v3s: Parent the CSI module clock to the ISP PLL
  • [PATCH 03/22] dt-bindings: sun6i-a31-mipi-dphy: Add optional direction property
  • [PATCH 04/22] phy: allwinner: phy-sun6i-mipi-dphy: Support D-PHY Rx mode for MIPI CSI-2
  • [PATCH 05/22] dt-bindings: media: sun6i-a31-csi: Add MIPI CSI-2 input port
  • [PATCH 06/22] dt-bindings: media: Add Allwinner A31 MIPI CSI-2 bindings documentation
  • [PATCH 07/22] media: sunxi: Add support for the A31 MIPI CSI-2 controller
  • [PATCH 08/22] MAINTAINERS: Add entry for the Allwinner A31 MIPI CSI-2 bridge driver
  • [PATCH 09/22] ARM: dts: sun8i: v3s: Add nodes for MIPI CSI-2 support
  • [PATCH 10/22] dt-bindings: media: Add Allwinner A83T MIPI CSI-2 bindings documentation
  • [PATCH 11/22] media: sunxi: Add support for the A83T MIPI CSI-2 controller
  • [PATCH 12/22] MAINTAINERS: Add entry for the Allwinner A83T MIPI CSI-2 bridge
  • [PATCH 13/22] ARM: dts: sun8i: a83t: Add MIPI CSI-2 controller node
  • [PATCH 14/22] ARM: dts: sun8i: a83t: bananapi-m3: Enable MIPI CSI-2 with OV8865
  • [PATCH 15/22] media: sunxi: Remove the sun6i-csi driver implementation
  • [PATCH 16/22] media: sunxi: Introduce a rewritten sun6i-csi driver
  • [PATCH 17/22] dt-bindings: media: Add Allwinner A31 ISP bindings documentation
  • [PATCH 18/22] dt-bindings: media: sun6i-a31-csi: Add ISP output port
  • [PATCH 19/22] soc: sunxi: mbus: Add A31 ISP compatibles to the list
  • [PATCH 20/22] staging: media: Add support for the Allwinner A31 ISP
  • [PATCH 21/22] MAINTAINERS: Add entry for the Allwinner A31 ISP driver
  • [PATCH 22/22] ARM: dts: sun8i: v3s: Add support for the ISP

Our currently proposed driver for the Allwinner ISP only supports a limited set of features: debayering with coefficients and 2D noise filtering. These features were sufficient for our use case, and allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator.

As the driver for now relies on a specific user-space API that does not yet cover all aspects of the ISP, the driver was submitted to the Linux kernel staging area and will probably stay there until all ISP features are properly described.

Our work on this advanced camera support, including the ISP driver, has been described in the talk we have given earlier this week at the Embedded Linux Conference, for which the slides are already available.

Additional features and future work

The Allwinner ISP supports a lot more features beyond just debayering and noise filtering. For example, it supports statistics to implement 3A algorithms (auto-focus, auto-exposition and auto-white-balance) which are necessary to avoid manual configuration of scene-specific parameters. These could typically be implemented in libcamera, the community free software project that supports complex image capture pipelines and ISPs.

As a result Bootlin would be very interested to continue this work and bring this driver to a more advanced state. So if you have a project that could help move this topic forward, do not hesitate to contact us about it!