The backbone of a Linux Industrial I/O driver

As part of recent projects, we had to dig into the Linux kernel Industrial I/O (IIO) subsystem with the goals of supporting a new ADC and adding new features to an existing driver. These tasks involved quite a few discussions between our engineering team and the IIO maintainers and reviewers. The aim of this blog post is to summarize the substance of these explanations to help others understand how an IIO kernel driver works and interacts with the core IIO subsystem.

Disclaimer: The IIO core is huge and keeps evolving. The aim of this article is not to cover it entirely, but at least explain our knowledge of how to use its basic features for common situations.

What is IIO?

The Industrial I/O subsystem covers any type of device that is commonly called as a “sensor”: ADCs, IMUs, temperature sensors, accelerometers, pressure sensors, potentiometers, light sensors, proximity sensors, etc (as well as few actuators, which I will on purpose disregard in this blog post). All these devices, besides measuring truly different physical components of our three dimensional world, end-up sharing quite a few properties. Any of these sensors must first be configured in order to know what must be measured and possibly how. When adequate, the device must be triggered in order to start converting. When the requested samples are ready, there must be some kind of signaling involved in order for the user to retrieve and process the data.

When thinking about the generic interfaces which could be needed by all these devices, it is quite straightforward to list:

  • The configuration before sampling
  • The triggering mechanism
  • The signaling for an end of conversion situation
  • The reading of the samples
  • The advertisement of the data

Registering an IIO device

The IIO core manipulates struct iio_dev * objects which inherits from struct device. This object should be allocated by the device driver with devm_iio_device_alloc(), providing the size of the driver’s internal structure as second argument. The allocated area dedicated for this internal pointer can be then retrieved with iio_priv().

This iio_dev structure must then be filled with a number of information:

  • The name of the device
  • A set of struct iio_info operations, typically a hook to read one or multiple samples on demand, optionally be able to write to the device, etc.
  • A set of supported modes, such as INDIO_DIRECT_MODE, which is used when samples can be retrieved at any time by the user from sysfs.
  • A scan mask, namely available_scan_masks which defines what are the possible/impossible scan combinations when requesting a read. Typically, a device might be configured to scan all of its internal channels from 1 to N. This can be described with a list of GENMASK(X, 0), with X ranging from 0 to the maximum number of channels. When the user will request a given set of channels, the IIO core will go through all the available masks registered by the driver and pick the first one that contains the desired channels. The selected mask will be available to the driver through the active_scan_mask entry of the iio_dev structure. If ‘anything goes’ and the devices has no restriction regarding which channel(s) can be scanned, this field should be skipped.
  • A definition of all the possible channels, including the type of physical measurements the device is able to perform (IIO_VOLTAGE, IIO_CURRENT, IIO_TEMPERATURE, IIO_STEPS, IIO_ROT, etc), the channel index and the data format.

Here is an example of channel description and below the meaning of these fields.

struct iio_chan_spec chan1 = {
	.type = IIO_VOLTAGE,
	.indexed = 0,
	.channel = index,
	.info_mask_separate = BIT(IIO_CHAN_INFO_RAW),
	.info_mask_shared_by_type = BIT(IIO_CHAN_INFO_SCALE),
	.scan_index = 1,
	.scan_type = {
		.sign = 'u',
		.realbits = 10,
		.storagebits = 16,
		.shift = 2,
		.endianness = IIO_BE,
	},
}
  • .info_mask_separate indicates an entry in sysfs that will be present for all the channels. IIO_CHAN_INFO_RAW is the raw value of the sample.
  • .info_mask_shared_by_type indicates an entry in sysfs that will be sorted by the type. IIO_CHAN_INFO_SCALE means that there will be a common voltage scale sysfs entry shared by all the voltage raw entries. If the device was also able to read a temperature, we would also get a single file indicating the scale for all the temperature samples.
  • The .scan_type field in the example indicates that values are provided as 16-bit big-endian samples that must be shifted by two bits. The full scale range is 0-1023. This conversion only applies to the buffer reading path: raw values directly read from sysfs and returned by the ->read_raw() hook (see below) should be converted by the driver itself.

Once the device fully described (and initialized, of course), the driver must register it with devm_iio_device_register().

Scaling factors

The int (*read_raw)(struct iio_dev *indio_dev, struct iio_chan_spec const *chan, int *val, int *val2, long mask) callback will be executed when reading either of the ‘raw value’ or ‘scale’ files from sysfs.

The type of data that must be returned is provided in the mask parameter: IIO_CHAN_INFO_RAW to retrieve the raw measurement or IIO_CHAN_INFO_SCALE to retrieve the scaling parameters, based on the scale information available in the iio_chan_spec structure that describes the channel.

For the IIO_CHAN_INFO_RAW case, most drivers return an IIO_VAL_INT type which can be simply “returned” into the *val argument. It is however possible to return a fixed point number, in this case the logic explained right after applies.

For the IIO_CHAN_INFO_SCALE case, the return value indicates what type of scaling should be done. In most cases here a fixed point value will be used so *val and *val2 will carry the scaling parameters. Here are two examples:

  1. IIO_TEMP example:
    *val = 1;
    *val2 = 8;
    ret = IIO_VAL_FRACTIONAL;
    

    The full scale sample value should be multiplied by 1/8 in order to get Celcius degrees.

  2. IIO_VOLTAGE example:

    *val = 2500;
    *val2 = 10;
    ret = IIO_VAL_FRACTIONAL_LOG2;
    

    The full scale range is a 10-bit value mapped to a 0-2500mV input level, said otherwise the scaling factor should be 2500 / 1024. The core will automatically do the computation of this factor and return 2,44140625 to the user in order to get milli-Volts.

Sampling

There are two basic common cases here.

In simple situations, a “single” on-demand read was issued by user-space directly by reading /sys/bus/devices/iio:device/in_<type><index>_raw. In this case the ->read_raw() callback should handle basically all the steps necessary to get a measurement, as detailed in our introduction.

However, user-space can also pick a more advanced way of interacting with the measurement device, called triggers.

A trigger is a specific configuration of the device which will sample a number of channels upon a specific event. This event might be the user requesting it from userspace with a so called software trigger, it might also be an external hardware event, or a periodic signal, or an internal continuous read mode… There are many ways of triggering a sensor and they are all covered by the subsystem.

Many devices cannot handle both modes at the same time. The only situation where this might work smoothly is when a device provides a hardware FIFO where you can read from (or a ‘latest value’ register) while not disrupting the FIFO read back. Otherwise, it will be needed, in order to avoid collisions between these two modes, to verify that exclusive access to the device is granted with a call to iio_device_claim_direct_mode() when starting a direct mode operation. As this helper grabs a mutex, it should be only called from process context and always be balanced with a call to iio_device_release_direct_mode().

IIO interoperability model

In the IIO core these four concepts are used:

  • IIO device: the hardware part which produces samples
  • IIO trigger: the signaling capability to request a conversion start
  • IIO buffers: where to store the samples
  • IIO events: threshold detectors

Even though a single hardware device might have hardware support for all these features, they must be described and handled separately so that, when applicable, other IIO devices might use them as well, eg. IIO device 2 could start a conversion upon IIO device 1 trigger state change. In practice it is not always possible but the way the API is built should lead us to keep things well separated anyway.

Registering a trigger

A struct iio_trigger must be allocated by devm_iio_trigger_alloc(), giving the new trigger a name.

The trigger should then receive a set of operations (struct iio_trigger_ops) with at least ->set_trigger_state() implemented, in order to switch on and off the trigger. One can use iio_trigger_set_drvdata() in order to link private data with the trigger and get this pointer back from the trigger callbacks.

Once initialized, devm_iio_trigger_register() will register the IIO trigger. This trigger will appear as a dedicated IIO device in sysfs.

It is likely that an IRQ will need to be registered as part of the trigger initialization step: the driver must be notified somehow that the trigger was toggled. If the asynchronous signaling is tied to a “trigger change” condition, which is the easiest situation, then it is advised to provide iio_trigger_generic_data_rdy_poll() as hard IRQ handler. This helper will just call iio_trigger_poll() and return.

You may of course want to handle more than this but in any case the rule is clear, triggers, buffers and devices should be fully separated. Hence, do not directly handle any data from this handler: an IIO trigger is only supposed to indicate a hardware transition, no more.

The call to iio_trigger_poll() will effectively go through the IIO internal interrupt tree, find the device that is connected to the trigger which fired and call the relevant handler in order to request the waiting device to process the data (which may be identical or different than the triggering device).

In the case of the device being limited to, for instance, an End Of Conversion (EOC) interrupt, you should still consider this signal as being suitable for being registered as a trigger. Yes, this might imply an additional delay between the hardware toggling and the IRQ being fired which is not ideal, but from a software point of view, the split between driver code and core logic will let other IIO devices use this IRQ as a trigger with no additional change needed to your code.

Note: There is one exception here. When a device does not provide any visible per-scan interrupt and the software has only access to some kind of FIFO watermark events, the whole trigger + buffer representation is swapped with a pure buffer-only implementation.

Registering triggered buffers

If the device itself is able to provide fast samples, the driver should also register a buffer, with iio_triggered_buffer_setup(). Both a hard IRQ handler and a threaded IRQ handler can be registered, as well as additional callbacks called before and after enabling and disabling the buffers in order to eg. configure the requested channels based on the current ->active_scan_mask.

Upon a trigger condition, these are the handlers that might be chosen by the core if the trigger is connected to your device!

The hard IRQ handler might be used to eg. save timestamps. The threaded IRQ handler is dedicated to the data processing. Depending of the type of trigger (iio_trigger_using_own()) the driver must decide whether it should start a conversion manually or if the data is waiting somewhere in a hardware FIFO, ready to be retrieved.

The final step is to push the samples into the core’s buffers. This should not be done manually. Let’s say that the user requested channels 0, 1 and 3 while the selected scan mask was including all channels from 0 to 4. Just calling iio_push_to_buffers() is the solution: the core knows that it will receive five samples of 16 bits, it also knows that the user only requested three of them and will automatically pick the right ones.

With all these IIO objects registered, you should be able to properly interact with the core and the other drivers, providing trigger capabilities to third party devices, or benefiting from other’s triggers.

What if my design lacks trigger capabilities?

You can still use triggers by enabling IIO_CONFIGFS (enables the configuration interface) and IIO_SW_TRIGGER. Then, you can either choose to trigger your scans from userspace with a simple file write, thanks to CONFIG_IIO_SYSFS_TRIGGER, or leverage timers to get periodic scans with CONFIG_IIO_HRTIMER_TRIGGER.

As an example, here is how to create a sysfs trigger:

# echo 0 > /sys/bus/iio/devices/iio_sysfs_trigger/add_trigger
# cat /sys/bus/iio/devices/iio_sysfs_trigger/trigger0/name
sysfstrig0

And here is how to create a timer based software trigger:

# mkdir -p /config
# mount -t configfs none /config
# mkdir /config/iio/triggers/hrtimer/my_5ms_trigger
# cat /sys/bus/iio/devices/trigger0/name
my_5ms_trigger
# echo 200 > /sys/bus/iio/devices/trigger0/sampling_frequency

How to use triggers and buffers from userspace

Just for the reference, linking an IIO trigger to an IIO device is as simple as:

# cat /sys/bus/iio/devices/trigger0/name > /sys/bus/iio/devices/iio:device0/trigger/current_trigger

The next step is to configure the channels that should be scanned:

# echo 1 > /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage0_en
# echo 1 > /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage1_en
# echo 1 > /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage3_en

Starting the sampling process is managed with:

# echo 1 > /sys/bus/iio/devices/iio:device0/buffer/enable

In the case of a sysfs software trigger, it is the user’s responsibility to timely run:

# echo 1 > /sys/bus/iio/devices/trigger0/trigger_now

The samples are available to be read in /dev/iio\:device0.

The decoding process before the scaling operation must be performed by the userspace, following the content of:

# cat /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage0_type
be:u12/16>>2 

Which in this case would mean that each sample is 16 bits wide, values should be considered big-endian, shifted twice before being considered as an unsigned 12-bit value.

# od -t x1 /dev/iio\:device0
0000000 08 06

Should be interpreted as 0x806 >> 2 = 0x201, which should be then multipled by the scaling factor in order to get the final mV value.

Conclusion

While contributing to this (relatively new) subsystem, we discovered a number of interesting features and design choices which would really benefit from a much tougher in-kernel documentation as most of the available information explains how to use IIO (with libiio or configfs) more than how to write a decent and properly shaped IIO device driver. As the subsystem is still pretty recent, it is valid to look at existing drivers to make design choices, but that is not a magic solution as no device never fully matches any software API anyway, and sensors unfortunately do not escape from that sticky rule.

We want to warmly thank Jonathan Cameron, IIO founder and maintainer, for his precious feedback on the mailing list, as well as his valuable review and contribution to this blog post.

We hope this article will help you go through this API and if it does, please mind letting us know by dropping a comment in the section down below!

On-going Bootlin contributions to the Video4Linux subsystem: camera, camera sensors, video encoding

Over the past years, we have been more and more involved in projects that have significant multimedia requirements. As part of this trend, 2020 has lead us to work on a number of contributions to the Video4Linux subsystem of the Linux kernel, with new drivers for camera interfaces, camera sensors, video decoders, and even HW-accelerated video encoding. In this blog post, we propose to summarize our contributions and their status on the following topics:

  • Rockchip PX30, RK1808, RK3128 and RK3288 camera interface driver
  • Allwinner A31, V3s/V3/S3 and A83T MIPI CSI-2 support for the camera interface driver
  • Omnivision OV8865 camera sensor driver
  • Omnivision OV5648 camera sensor driver
  • TW9900 PAL/NTSC video decoder driver
  • Rockchip HW-accelerated H264 video encoding

Rockchip camera interface

Rockchip camera interfaceThe Rockchip ARM processors are known to have very good support in the upstream Linux kernel. However, one area where the support was lacking is in the support of the camera interface used by those SoCs. And it turns out that Bootlin engineer Maxime Chevallier has worked precisely on this topic throughout 2020: the development and upstreaming of the rkvip driver, a Video4Linux driver for the Rockchip camera interface. While the work was done and tested on a Rockchip PX30 platform, the same camera interface is used on RK1808, RK3128 and RK3288.

Several iterations of the driver have been posted on the linux-media mailing list, with the latest iteration, version 5, posted on December 29, 2020:

Maxime Chevallier (3):
  media: dt-bindings: media: Document Rockchip VIP bindings
  media: rockchip: Introduce driver for Rockhip's camera interface
  arm64: dts: rockchip: Add the camera interface description of the PX30

 .../bindings/media/rockchip-vip.yaml          |  101 ++
 arch/arm64/boot/dts/rockchip/px30.dtsi        |   12 +
 drivers/media/platform/Kconfig                |   15 +
 drivers/media/platform/Makefile               |    1 +
 drivers/media/platform/rockchip/vip/Makefile  |    3 +
 drivers/media/platform/rockchip/vip/capture.c | 1146 +++++++++++++++++
 drivers/media/platform/rockchip/vip/dev.c     |  331 +++++
 drivers/media/platform/rockchip/vip/dev.h     |  203 +++
 drivers/media/platform/rockchip/vip/regs.h    |  260 ++++
 9 files changed, 2072 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-vip.yaml
 create mode 100644 drivers/media/platform/rockchip/vip/Makefile
 create mode 100644 drivers/media/platform/rockchip/vip/capture.c
 create mode 100644 drivers/media/platform/rockchip/vip/dev.c
 create mode 100644 drivers/media/platform/rockchip/vip/dev.h
 create mode 100644 drivers/media/platform/rockchip/vip/regs.h

We’re hoping to get this driver merged soon, as we have now addressed the feedback that was received through the 5 iterations the patch series as gone through. It should be noted that for now it only supports the parallel BT656 interface as this is what we needed for our current project, we are definitely able to extend it to support MIPI CSI2 as well if you’re interested!

It should be noted that as a result of this work, Maxime Chevallier also prepared and delivered a talk From a video sensor to your display which was given at the Embedded Linux Conference Europe 2020. See the slides and video.

Allwinner MIPI CSI2 camera interface

Allwinner MIPI CSI2As part of an internship in 2020 and then a customer project, Bootlin intern Kévin L’Hôpital and Bootlin engineer Paul Kocialkowski worked on extending the Allwinnera camera interface support with support for MIPI CSI2 cameras. In fact, this addition was done to two Allwinner camera interface drivers: the sun6i driver which is used on Allwinner A31 and V3s/V3/S3, and the sun8i-a83t, which is used on the Allwinner A83T.

Through a fairly long 15 patches patch series, support for MIPI CSI2 is added to both camera interface controllers. We have tested both with Omnivision sensors, which are described below.

The series is currently in its third iteration, which was posted by Paul Kocialkowski on December 11, 2020 on the linux-media mailing list:


Paul Kocialkowski (15):
  docs: phy: Add a part about PHY mode and submode
  phy: Distinguish between Rx and Tx for MIPI D-PHY with submodes
  phy: allwinner: phy-sun6i-mipi-dphy: Support D-PHY Rx mode for MIPI
    CSI-2
  media: sun6i-csi: Use common V4L2 format info for storage bpp
  media: sun6i-csi: Only configure the interface data width for parallel
  dt-bindings: media: sun6i-a31-csi: Add MIPI CSI-2 input port
  media: sun6i-csi: Add support for MIPI CSI-2 bridge input
  dt-bindings: media: Add A31 MIPI CSI-2 bindings documentation
  media: sunxi: Add support for the A31 MIPI CSI-2 controller
  ARM: dts: sun8i: v3s: Add nodes for MIPI CSI-2 support
  MAINTAINERS: Add entry for the Allwinner A31 MIPI CSI-2 bridge
  dt-bindings: media: Add A83T MIPI CSI-2 bindings documentation
  media: sunxi: Add support for the A83T MIPI CSI-2 controller
  ARM: dts: sun8i: a83t: Add MIPI CSI-2 controller node
  MAINTAINERS: Add entry for the Allwinner A83T MIPI CSI-2 bridge

 .../media/allwinner,sun6i-a31-csi.yaml        |  88 ++-
 .../media/allwinner,sun6i-a31-mipi-csi2.yaml  | 149 ++++
 .../media/allwinner,sun8i-a83t-mipi-csi2.yaml | 147 ++++
 Documentation/driver-api/phy/phy.rst          |  18 +
 MAINTAINERS                                   |  16 +
 arch/arm/boot/dts/sun8i-a83t-bananapi-m3.dts  |   2 +-
 arch/arm/boot/dts/sun8i-a83t.dtsi             |  26 +
 arch/arm/boot/dts/sun8i-v3s.dtsi              |  67 ++
 drivers/media/platform/sunxi/Kconfig          |   2 +
 drivers/media/platform/sunxi/Makefile         |   2 +
 .../platform/sunxi/sun6i-csi/sun6i_csi.c      | 165 +++--
 .../platform/sunxi/sun6i-csi/sun6i_csi.h      |  58 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |  53 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.h    |   7 +-
 .../platform/sunxi/sun6i-mipi-csi2/Kconfig    |  12 +
 .../platform/sunxi/sun6i-mipi-csi2/Makefile   |   4 +
 .../sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.c   | 590 ++++++++++++++++
 .../sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.h   | 117 ++++
 .../sunxi/sun8i-a83t-mipi-csi2/Kconfig        |  11 +
 .../sunxi/sun8i-a83t-mipi-csi2/Makefile       |   4 +
 .../sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.c    |  92 +++
 .../sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.h    |  39 ++
 .../sun8i_a83t_mipi_csi2.c                    | 657 ++++++++++++++++++
 .../sun8i_a83t_mipi_csi2.h                    | 197 ++++++
 drivers/phy/allwinner/phy-sun6i-mipi-dphy.c   | 164 ++++-
 drivers/staging/media/rkisp1/rkisp1-isp.c     |   3 +-
 include/linux/phy/phy-mipi-dphy.h             |  13 +
 27 files changed, 2581 insertions(+), 122 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/allwinner,sun6i-a31-mipi-csi2.yaml
 create mode 100644 Documentation/devicetree/bindings/media/allwinner,sun8i-a83t-mipi-csi2.yaml
 create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/Kconfig
 create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/Makefile
 create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.c
 create mode 100644 drivers/media/platform/sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.h
 create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/Kconfig
 create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/Makefile
 create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.c
 create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_dphy.h
 create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_mipi_csi2.c
 create mode 100644 drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/sun8i_a83t_mipi_csi2.h

Here as well, the patch series has gone through a number of iterations, with significant reshaping to take into account the comments and feedback of other kernel developers and maintainers, so we hope to be near the point where it can be merged.

Omnivision OV8865 camera sensor driver

OV8865 block diagramAs part of his internship at Bootlin in 2020, Kévin L’Hôpital implemented a driver for the OV8865 camera sensor, connected over MIPI CSI2 to an Allwinner A83T platform. This OV8865 was then taken by Bootlin engineer Paul Kocialkowski, who did additional rework and polishing.

We are currently at the 4th iteration of this driver, which has been posted on December 11, 2020, and it has now been accepted and submitted to the V4L maintainer in a pull request.


Kévin L'hôpital (1):
  ARM: dts: sun8i: a83t: bananapi-m3: Enable MIPI CSI-2 with OV8865

Paul Kocialkowski (2):
  dt-bindings: media: i2c: Add OV8865 bindings documentation
  media: i2c: Add support for the OV8865 image sensor

 .../bindings/media/i2c/ovti,ov8865.yaml       |  124 +
 arch/arm/boot/dts/sun8i-a83t-bananapi-m3.dts  |  102 +
 drivers/media/i2c/Kconfig                     |   13 +
 drivers/media/i2c/Makefile                    |    1 +
 drivers/media/i2c/ov8865.c                    | 2981 +++++++++++++++++
 5 files changed, 3221 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/i2c/ovti,ov8865.yaml
 create mode 100644 drivers/media/i2c/ov8865.c

Omnivision OV5648 camera sensor driver

OV5648 block diagramIn addition to the work done by Bootlin intern Kévin L’Hôpital on OV8865 with Allwinner A83T, Paul Kocialkowski worked on OV5648 with Allwinner V3s, also connected over MIPI CSI2. This work results in a driver for the OV5648 camera sensor, which Paul has submitted to the linux-media mailing list.

This driver is now in is 5th iteration, posted on December 11, 2020, and it has now been accepted and submitted to the V4L maintainer in a pull request.


Paul Kocialkowski (2):
  dt-bindings: media: i2c: Add OV5648 bindings documentation
  media: i2c: Add support for the OV5648 image sensor

 .../bindings/media/i2c/ovti,ov5648.yaml       |  115 +
 drivers/media/i2c/Kconfig                     |   13 +
 drivers/media/i2c/Makefile                    |    1 +
 drivers/media/i2c/ov5648.c                    | 2638 +++++++++++++++++
 4 files changed, 2767 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/i2c/ovti,ov5648.yaml
 create mode 100644 drivers/media/i2c/ov5648.c

TW9900 PAL/NTSC video decoder driver

TW9900In addition to working on the Rockchip camera interface driver, Maxime Chevallier has also worked on a driver for the TW9900 PAL/NTSC video decoder. This chip from Renesas, takes as input an analog PAL or NTSC signal, digitizes it and outputs it on a parallel BT656 interface, which in our case was connected to a Rockchip PX30 platform.

Maxime posted the third iteration of the patch series adding this driver on December 22, 2020 on the linux-media mailing list.

Maxime Chevallier (3):
  dt-bindings: vendor-prefixes: Add techwell vendor prefix
  media: dt-bindings: media: i2c: Add bindings for TW9900
  media: i2c: Introduce a driver for the Techwell TW9900 decoder

 .../devicetree/bindings/media/i2c/tw9900.yaml |  60 ++
 .../devicetree/bindings/vendor-prefixes.yaml  |   2 +
 MAINTAINERS                                   |   6 +
 drivers/media/i2c/Kconfig                     |  11 +
 drivers/media/i2c/Makefile                    |   1 +
 drivers/media/i2c/tw9900.c                    | 617 ++++++++++++++++++
 6 files changed, 697 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/i2c/tw9900.yaml
 create mode 100644 drivers/media/i2c/tw9900.c

Rockchip HW-accelerated H264 video encoding

In 2018 and thanks to success of the crowd-funding campaign we ran back then, Bootlin engineer Paul Kocialkowski pioneered support for stateless video decoders in the Linux kernel, with a first driver supporting MPEG2, H264 and H265 HW-accelerated video decoding on Allwinner platforms.

Rockchip video encoderIn 2020, Paul was tasked to work on HW-accelerated H264 video encoding for Rockchip platforms, which also use a stateless video encoder. Of course, Paul took the same approach of going towards an upstream-acceptable solution rather than relying on out-of-tree and vendor-specific solutions provided by Rockchip.

Paul has been able to implement a working solution for one of our customers, and while the result is not yet in a shape where it can be submitted upstream, Paul has presented its result at the Embedded Linux Conference Europe 2020: the slides and video. The kernel code is available at https://github.com/bootlin/linux/tree/hantro/h264-encoding while the user-space code is available at https://github.com/bootlin/v4l2-hantro-h264-encoder.

As explained in Paul’s talk, this is not fully ready for upstream, as lots of discussions are needed on the user-space APIs, especially around the topic of rate control.

If you are interested in having this work fully available in the upstream Linux kernel, please contact us. We are looking for additional funding and support to push this completely upstream.

Conclusion

As can be seen from the numerous topics covered in this blog post, Bootlin has significant experience with the Video4Linux subsystem, and is able to both implement support for new hardware, extend the Video4Linux subsystem if needed, and contribute drivers and changes to the official Linux kernel.