Over the past year Bootlin engineer Luca Ceresoli has been working to add a device driver for the parallel camera interface of the NVIDIA Tegra20 System on Chip into the mainline Linux kernel.
The main challenge faced during this work has been the lack of documentation. So the work has been based on a driver from an NVIDIA BSP, forked from a 3.1 kernel (which has been released back in 2011!). The old driver code base needed a huge rework, being largely rewritten, and not only because of the changes in 10+ years of kernel development.
The mainline kernel already has a driver for CSI capture on Tegra210, albeit in staging. The two hardware components have some common functionality, thus to avoid code duplication Luca augmented the existing driver and generalized the code implementing common areas instead of adding a new driver. This posed the additional challenge of not breaking functionality on another SoC, based on a different architecture and using a different video bus… all without access to such other hardware!
If you have a device using Tegra20 parallel capture or Tegra210 CSI video capture, this is a great opportunity to test the code and report your findings! And in case you don’t have the hardware, you’d still be very welcome in reviewing the patches.
Finally, if you have access to the Tegra20 documentation, we’d love to know: the driver could possibly be improved with good knowledge of the hardware.
Since then we had the opportunity to take things further and start tackling the next steps for advanced camera support in mainline Linux on Allwinner SoCs!
With MIPI CSI-2 support and proper sensor drivers available in V4L2, we were able to capture raw bayer data provided by the sensors. But this data does not constitute the final picture that can be displayed or encoded into a file: a number of enhancement and transformation steps are required to achieve a visually-pleasing result that users typically expect.
These steps are quite calculation-intensive and it does not make sense to implement them with a software pipeline, especially with rates of 25, 30 or even 60 frames per second that are typically expected for video recording.
An open-source and upstream driver for the Allwinner ISP
Allwinner SoCs that support MIPI CSI-2 also include an Image Signal Processor hardware unit, a dedicated accelerator for enhancing and transforming raw data received from sensors.
Features of the ISP as described in the Allwinner V3 datasheet.
Since support for this ISP was implemented using a non-free blob in Allwinner SDKs, this area remained unsupported in mainline Linux… Until now!
Thanks to some help from Allwinner, we were able to implement a proper V4L2 driver for the Allwinner ISP found in the Allwinner V3, completely open-source, with no binary blob involved. This work was recently submitted upstream, with a first revision totaling more than 8000 new lines of code, which comes together with a significant rework of the Allwinner camera interface driver to make it usable with or without the ISP, and including the new MIPI CSI-2 support which we had submitted previously. We are very happy to keep contributing to advancing fully open-source Allwinner SoCs support in mainline Linux and help tackle some of the remaining areas there!
Our currently proposed driver for the Allwinner ISP only supports a limited set of features: debayering with coefficients and 2D noise filtering. These features were sufficient for our use case, and allowed to offload the computationally intensive debayering process to a dedicated hardware accelerator.
As the driver for now relies on a specific user-space API that does not yet cover all aspects of the ISP, the driver was submitted to the Linux kernel staging area and will probably stay there until all ISP features are properly described.
Our work on this advanced camera support, including the ISP driver, has been described in the talk we have given earlier this week at the Embedded Linux Conference, for which the slides are already available.
Additional features and future work
The Allwinner ISP supports a lot more features beyond just debayering and noise filtering. For example, it supports statistics to implement 3A algorithms (auto-focus, auto-exposition and auto-white-balance) which are necessary to avoid manual configuration of scene-specific parameters. These could typically be implemented in libcamera, the community free software project that supports complex image capture pipelines and ISPs.
As a result Bootlin would be very interested to continue this work and bring this driver to a more advanced state. So if you have a project that could help move this topic forward, do not hesitate to contact us about it!
Over the past years, we have been more and more involved in projects that have significant multimedia requirements. As part of this trend, 2020 has lead us to work on a number of contributions to the Video4Linux subsystem of the Linux kernel, with new drivers for camera interfaces, camera sensors, video decoders, and even HW-accelerated video encoding. In this blog post, we propose to summarize our contributions and their status on the following topics:
Rockchip PX30, RK1808, RK3128 and RK3288 camera interface driver
Allwinner A31, V3s/V3/S3 and A83T MIPI CSI-2 support for the camera interface driver
Omnivision OV8865 camera sensor driver
Omnivision OV5648 camera sensor driver
TW9900 PAL/NTSC video decoder driver
Rockchip HW-accelerated H264 video encoding
Rockchip camera interface
The Rockchip ARM processors are known to have very good support in the upstream Linux kernel. However, one area where the support was lacking is in the support of the camera interface used by those SoCs. And it turns out that Bootlin engineer Maxime Chevallier has worked precisely on this topic throughout 2020: the development and upstreaming of the rkvip driver, a Video4Linux driver for the Rockchip camera interface. While the work was done and tested on a Rockchip PX30 platform, the same camera interface is used on RK1808, RK3128 and RK3288.
Several iterations of the driver have been posted on the linux-media mailing list, with the latest iteration, version 5, posted on December 29, 2020:
We’re hoping to get this driver merged soon, as we have now addressed the feedback that was received through the 5 iterations the patch series as gone through. It should be noted that for now it only supports the parallel BT656 interface as this is what we needed for our current project, we are definitely able to extend it to support MIPI CSI2 as well if you’re interested!
It should be noted that as a result of this work, Maxime Chevallier also prepared and delivered a talk From a video sensor to your display which was given at the Embedded Linux Conference Europe 2020. See the slides and video.
Allwinner MIPI CSI2 camera interface
As part of an internship in 2020 and then a customer project, Bootlin intern Kévin L’Hôpital and Bootlin engineer Paul Kocialkowski worked on extending the Allwinnera camera interface support with support for MIPI CSI2 cameras. In fact, this addition was done to two Allwinner camera interface drivers: the sun6i driver which is used on Allwinner A31 and V3s/V3/S3, and the sun8i-a83t, which is used on the Allwinner A83T.
Through a fairly long 15 patches patch series, support for MIPI CSI2 is added to both camera interface controllers. We have tested both with Omnivision sensors, which are described below.
The series is currently in its third iteration, which was posted by Paul Kocialkowski on December 11, 2020 on the linux-media mailing list:
Paul Kocialkowski (15):
docs: phy: Add a part about PHY mode and submode
phy: Distinguish between Rx and Tx for MIPI D-PHY with submodes
phy: allwinner: phy-sun6i-mipi-dphy: Support D-PHY Rx mode for MIPI
CSI-2
media: sun6i-csi: Use common V4L2 format info for storage bpp
media: sun6i-csi: Only configure the interface data width for parallel
dt-bindings: media: sun6i-a31-csi: Add MIPI CSI-2 input port
media: sun6i-csi: Add support for MIPI CSI-2 bridge input
dt-bindings: media: Add A31 MIPI CSI-2 bindings documentation
media: sunxi: Add support for the A31 MIPI CSI-2 controller
ARM: dts: sun8i: v3s: Add nodes for MIPI CSI-2 support
MAINTAINERS: Add entry for the Allwinner A31 MIPI CSI-2 bridge
dt-bindings: media: Add A83T MIPI CSI-2 bindings documentation
media: sunxi: Add support for the A83T MIPI CSI-2 controller
ARM: dts: sun8i: a83t: Add MIPI CSI-2 controller node
MAINTAINERS: Add entry for the Allwinner A83T MIPI CSI-2 bridge
.../media/allwinner,sun6i-a31-csi.yaml | 88 ++-
.../media/allwinner,sun6i-a31-mipi-csi2.yaml | 149 ++++
.../media/allwinner,sun8i-a83t-mipi-csi2.yaml | 147 ++++
Documentation/driver-api/phy/phy.rst | 18 +
MAINTAINERS | 16 +
arch/arm/boot/dts/sun8i-a83t-bananapi-m3.dts | 2 +-
arch/arm/boot/dts/sun8i-a83t.dtsi | 26 +
arch/arm/boot/dts/sun8i-v3s.dtsi | 67 ++
drivers/media/platform/sunxi/Kconfig | 2 +
drivers/media/platform/sunxi/Makefile | 2 +
.../platform/sunxi/sun6i-csi/sun6i_csi.c | 165 +++--
.../platform/sunxi/sun6i-csi/sun6i_csi.h | 58 +-
.../platform/sunxi/sun6i-csi/sun6i_video.c | 53 +-
.../platform/sunxi/sun6i-csi/sun6i_video.h | 7 +-
.../platform/sunxi/sun6i-mipi-csi2/Kconfig | 12 +
.../platform/sunxi/sun6i-mipi-csi2/Makefile | 4 +
.../sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.c | 590 ++++++++++++++++
.../sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.h | 117 ++++
.../sunxi/sun8i-a83t-mipi-csi2/Kconfig | 11 +
.../sunxi/sun8i-a83t-mipi-csi2/Makefile | 4 +
.../sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.c | 92 +++
.../sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.h | 39 ++
.../sun8i_a83t_mipi_csi2.c | 657 ++++++++++++++++++
.../sun8i_a83t_mipi_csi2.h | 197 ++++++
drivers/phy/allwinner/phy-sun6i-mipi-dphy.c | 164 ++++-
drivers/staging/media/rkisp1/rkisp1-isp.c | 3 +-
include/linux/phy/phy-mipi-dphy.h | 13 +
27 files changed, 2581 insertions(+), 122 deletions(-)
create mode 100644 Documentation/devicetree/bindings/media/allwinner,sun6i-a31-mipi-csi2.yaml
create mode 100644 Documentation/devicetree/bindings/media/allwinner,sun8i-a83t-mipi-csi2.yaml
create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/Kconfig
create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/Makefile
create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.c
create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.h
create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/Kconfig
create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/Makefile
create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.c
create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.h
create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_mipi_csi2.c
create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_mipi_csi2.h
Here as well, the patch series has gone through a number of iterations, with significant reshaping to take into account the comments and feedback of other kernel developers and maintainers, so we hope to be near the point where it can be merged.
Omnivision OV8865 camera sensor driver
As part of his internship at Bootlin in 2020, Kévin L’Hôpital implemented a driver for the OV8865 camera sensor, connected over MIPI CSI2 to an Allwinner A83T platform. This OV8865 was then taken by Bootlin engineer Paul Kocialkowski, who did additional rework and polishing.
We are currently at the 4th iteration of this driver, which has been posted on December 11, 2020, and it has now been accepted and submitted to the V4L maintainer in a pull request.
In addition to the work done by Bootlin intern Kévin L’Hôpital on OV8865 with Allwinner A83T, Paul Kocialkowski worked on OV5648 with Allwinner V3s, also connected over MIPI CSI2. This work results in a driver for the OV5648 camera sensor, which Paul has submitted to the linux-media mailing list.
This driver is now in is 5th iteration, posted on December 11, 2020, and it has now been accepted and submitted to the V4L maintainer in a pull request.
In addition to working on the Rockchip camera interface driver, Maxime Chevallier has also worked on a driver for the TW9900 PAL/NTSC video decoder. This chip from Renesas, takes as input an analog PAL or NTSC signal, digitizes it and outputs it on a parallel BT656 interface, which in our case was connected to a Rockchip PX30 platform.
Maxime posted the third iteration of the patch series adding this driver on December 22, 2020 on the linux-media mailing list.
In 2018 and thanks to success of the crowd-funding campaign we ran back then, Bootlin engineer Paul Kocialkowski pioneered support for stateless video decoders in the Linux kernel, with a first driver supporting MPEG2, H264 and H265 HW-accelerated video decoding on Allwinner platforms.
In 2020, Paul was tasked to work on HW-accelerated H264 video encoding for Rockchip platforms, which also use a stateless video encoder. Of course, Paul took the same approach of going towards an upstream-acceptable solution rather than relying on out-of-tree and vendor-specific solutions provided by Rockchip.
As explained in Paul’s talk, this is not fully ready for upstream, as lots of discussions are needed on the user-space APIs, especially around the topic of rate control.
If you are interested in having this work fully available in the upstream Linux kernel, please contact us. We are looking for additional funding and support to push this completely upstream.
Conclusion
As can be seen from the numerous topics covered in this blog post, Bootlin has significant experience with the Video4Linux subsystem, and is able to both implement support for new hardware, extend the Video4Linux subsystem if needed, and contribute drivers and changes to the official Linux kernel.
