SFP modules on a board running Linux

We recently worked on Linux support for a custom hardware platform based on the Texas Instruments AM335x system-on-chip, with a somewhat special networking setup: each of the two ports of the AM335x Ethernet MAC was connected to a Microchip VSC8572 Ethernet PHY, which itself allowed to access an SFP cage. In addition, the I2C buses connected to the SFP cages, which are used at runtime to communicate with the inserted SFP modules, instead of being connected to an I2C controller of the system-on-chip as they usually are, where connected to the I2C controller embedded in the VSC8572 PHYs.

The below diagram depicts the overall hardware layout:

Our goal was to use Linux and to offer runtime dynamic reconfiguration of the networking links based the SFP module plugged in. To achieve this we used, and extended, a combination of Linux kernel internal frameworks such as Phylink or the SFP bus support; and of networking device drivers. In this blog post, we’ll share some background information about these technologies, the challenges we faced and our current status.

Introduction to the SFP interface

SFP moduleThe small form-factor pluggable (SFP) is a hot-pluggable network interface module. Its electrical interface and its form-factor are well specified, which allows industry players to build platforms that can host SFP modules, and be sure that they will be able to use any available SFP module on the market. It is commonly used in the networking industry as it allows connecting various types of transceivers to a fixed interface.

A SFP cage provides in addition to data signals a number of control signals:

  • a Tx_Fault pin, for transmitter fault indication
  • a Tx_Disable pin, for disabling optical output
  • a MOD_Abs pin, to detect the absence of a module
  • an Rx_LOS pin, to denote a receiver loss of signal
  • a 2-wire data and clock lines, used to communicate with the modules

Modules plugged into SFP cages can be direct attached cables, in which case they do not have any built-in transceiver, or they can include a transceiver (i.e an embedded PHY), which transforms the signal into another format. This means that in our setup, there can be two PHYs between the Ethernet MAC and the physical medium: the Microchip VSC8572 PHY and the PHY embedded into the SFP module that is plugged in.

All SFP modules embed an EEPROM, accessible at a standardized I2C address and with a standardized format, which allows the host system to discover which SFP modules are connected what are their capabilities. In addition, if the SFP modules contains an embedded PHY, it is also accessible through the same I2C bus.

Challenges

We had to overcome a few challenges to get this setup working, using a mainline Linux kernel.

As we discussed earlier, having SFP modules meant the whole MAC-PHY-SFP link has to be reconfigured at runtime, as the PHY in the SFP module is hot-pluggable. To solve this issue a framework called Phylink, was introduced in mid-2017 to represent networking links and allowing their component to share states and to be reconfigured at runtime. For us, this meant we had to first convert the CPSW MAC driver to use this phylink framework. For a detailed explanation of what composes Ethernet links and why Phylink is needed, we gave a talk at the Embedded Linux Conference Europe in 2018. While we were working on this and after we first moved the CPSW MAC driver to use Phylink, this driver was rewritten and a new CPSW MAC driver was sent upstream (CONFIG_TI_CPSW vs CONFIG_TI_CPSW_SWITCHDEV). We are still using the old driver for now, and this is why we did not send our patches upstream as we think it does not make sense to convert a driver which is now deprecated.

A second challenge was to integrate the 2-wire capability of the VSC8572 PHY into the networking PHY and SFP common code, as our SFP modules I2C bus is connected to the PHY and not an I2C controller from the system-on-chip. We decided to expose this PHY 2-wire capability as an SMBus controller, as the functionality offered by the PHY does not make it a fully I2C compliant controller.

Outcome

The challenges described above made the project quite complex overall, but we were able to get SFP modules working, and to dynamically switch modes depending on the capabilities of the one currently plugged-in. We tested with both direct attached cables and a wide variety of SFP modules of different speeds and functionality. At the moment only a few patches were sent upstream, but we’ll contribute more over time.

For an overview of some of the patches we made and used, we pushed a branch on Github (be aware those patches aren’t upstream yet and they will need some further work to be acceptable upstream). Here is the details of the patches:

In terms of Device Tree representation, we first have a description of the two SFP cages. They describe the different GPIOs used for the control signals, as well as the I2C bus that goes to each SFP cage. Note that the gpio_sfp is a GPIO expander, itself on I2C, rather than directly GPIOs of the system-on-chip.

/ {
       sfp_eth0: sfp-eth0 {
               compatible = "sff,sfp";
               i2c-bus = <&phy0>;
               los-gpios = <&gpio_sfp 3 GPIO_ACTIVE_HIGH>;
               mod-def0-gpios = <&gpio_sfp 4 GPIO_ACTIVE_LOW>;
               tx-disable-gpios = <&gpio_sfp 5 GPIO_ACTIVE_HIGH>;
               tx-fault-gpios = <&gpio_sfp 6 GPIO_ACTIVE_HIGH>;
       };

       sfp_eth1: sfp-eth1 {
               compatible = "sff,sfp";
               i2c-bus = <&phy1>;
               los-gpios = <&gpio_sfp 10 GPIO_ACTIVE_HIGH>;
               mod-def0-gpios = <&gpio_sfp 11 GPIO_ACTIVE_LOW>;
               tx-disable-gpios = <&gpio_sfp 13 GPIO_ACTIVE_HIGH>;
               tx-fault-gpios  = <&gpio_sfp 12 GPIO_ACTIVE_HIGH>;
       };
};

Then the MAC is described as follows:

&mac {
      pinctrl-names = "default";
       pinctrl-0 = <&cpsw_default>;
       status = "okay";
       dual_emac;
};

&cpsw_emac0 {
       status = "okay";
       phy = <&phy0>;
       phy-mode = "rgmii-id";
       dual_emac_res_vlan = <1>;
};

&cpsw_emac1 {
       status = "okay";
       phy = <&phy1>;
       phy-mode = "rgmii-id";
       dual_emac_res_vlan = <2>;
};

So we have both ports of the MAC enabled with a RGMII interface to the PHY. And finally the MDIO bus of the system-on-chip is described as follows. We have two sub-nodes, one for each VSC8572 PHY, respectively at address 0x0 and 0x1 on the CPSW MDIO bus. Each PHY is connected to its respective SFP cage node (sfp_eth0 and sfp_eth1) and provides access to the SFP EEPROM as regular EEPROMs.

&davinci_mdio {
       pinctrl-names = "default";
       pinctrl-0 = <&davinci_mdio_default>;
       status = "okay";

       phy0: ethernet-phy@0 {
               #address-cells = <1>;
               #size-cells = <0>;

               reg = <0>;
               fiber-mode;
               vsc8584,los-active-low;
               sfp = <&sfp_eth0>;

               sfp0_eeprom: eeprom@50 {
                       compatible = "atmel,24c02";
                       reg = <0x50>;
                       read-only;
               };

               sfp0_eeprom_ext: eeprom@51 {
                       compatible = "atmel,24c02";
                       reg = <0x51>;
                       read-only;
               };
       };

       phy1: ethernet-phy@1 {
               #address-cells = <1>;
               #size-cells = <0>;

               reg = <1>;
               fiber-mode;
               vsc8584,los-active-low;
               sfp = <&sfp_eth1>;

               sfp1_eeprom: eeprom@50 {
                       compatible = "atmel,24c02";
                       reg = <0x50>;
                       read-only;
               };

               sfp1_eeprom_ext: eeprom@51 {
                       compatible = "atmel,24c02";
                       reg = <0x51>;
                       read-only;
               };
       };
};

Conclusion

While we are still working on pushing all of this work upstream, we’re happy to have been able to work on these topics. Do not hesitate to reach out of to us if you have projects that involve Linux and SFP modules!

Linux 5.6, Bootlin contributions inside

Linux 5.6 was released last Sunday. As usual, LWN has the best coverage of the new features merged in this release: part 1 and part 2. Sadly, the corresponding KernelNewbies page has not yet been updated with the usual very interesting summary of the important changes.

Bootlin contributed a total of 95 patches to this release, which makes us the 27th contributing company by number of commits, according to the statistics. The main highlights of our contributions are:

  • Our work on supporting hardware-offloading of MACsec encryption/decryption in the networking subsystem and support for this offloading for some Microchip/Vitesse PHYs has been merged. See our previous blog post for more details about this work done by Bootlin engineer Antoine Ténart
  • As part of our work on the Rockchip PX30 system-on-chip, we contributed support for LVDS display on Rockchip PX30, and support for the Satoz SAT050AT40H12R2 panel. This work was done by Miquèl Raynal
  • Alexandre Belloni as the RTC maintainer did his usual number of cleanup and improvements to existing RTC drivers
  • We did a number of small contributions to the Microchip AT91/SAMA5 platform: support for the Smartkiz platform from Overkiz, phylink improvements in the macb driver, etc.
  • Paul Kocialkowski improved the Intel GMA 500 DRM driver to support page flip.
  • Paul Kocialkowski contributed support for the Xylon LogiCVC GPIO controller, which is a preliminary step to contributing the Xylon LogiCVC display controller support. See our blog post on this topic.

In addition to being contributors, a number of Bootlin engineers are also maintainers of various parts of the Linux kernel, and as such:

  • Alexandre Belloni, as the RTC subsystem maintainer and Microchip platforms co-maintainer, has reviewed and merged 55 patches from other contributors
  • Miquèl Raynal, as the MTD co-maintainer, has reviewed and merged 21 patches from other contributors
  • Grégory Clement, as the Marvell EBU platform co-maintainer, has reviewed and merged 12 patches from other contributors

Here is the detail of all our contributions:

Covid-19: Bootlin proposes online sessions for all its courses

Tux working on embedded Linux on a couchLike most of us, due to the Covid-19 epidemic, you may be forced to work from home. To take advantage from this time confined at home, we are now proposing all our training courses as online seminars. You can then benefit from the contents and quality of Bootlin training sessions, without leaving the comfort and safety of your home. During our online seminars, our instructors will alternate between presentations and practical demonstrations, executing the instructions of our practical labs.

At any time, participants will be able to ask questions.

We can propose such remote training both through public online sessions, open to individual registration, as well as dedicated online sessions, for participants from the same company.

Public online sessions

We’re trying to propose time slots that should be manageable from Europe, Middle East, Africa and at least for the East Coast of North America. All such sessions will be taught in English. As usual with all our sessions, all our training materials (lectures and lab instructions) are freely available from the pages describing our courses.

Our Embedded Linux and Linux kernel courses are delivered over 7 half days of 4 hours each, while our Yocto Project, Buildroot and Linux Graphics courses are delivered over 4 half days of 4 hours each.

Here are all the available sessions. If the situation lasts longer, we will create new sessions as needed:

Type Dates Time Duration Expected trainer Cost and registration
Embedded Linux (agenda) Apr. 28, 29, 30, May 4, 5, 6, 7, 2020 14:00 -18:00 (Paris time) 28 h Michael Opdenacker 790 EUR + VAT* + fees
Linux kernel (agenda) May 11, 12, 18, 19, 25, 28, Jun. 2, 2020 14:00 – 18:00 (Paris time) 28 h Grégory Clement 790 EUR + VAT* + fees
Yocto Project (agenda) Jun. 11, 12, 16, 17, 2020 14:00 – 18:00 (Paris time) 16 h Alexandre Belloni 490 EUR + VAT* + fees
Buildroot (agenda) Jun. 4, 5, 8, 9, 2020 14:00 – 18:00 (Paris time) 16 h Thomas Petazzoni 490 EUR + VAT* + fees
Linux Graphics (agenda) Jun. 23, 24, 25, 26, 2020 14:00 – 18:00 (Paris time) 16 h Paul Kocialkowski 490 EUR + VAT* + fees

* VAT: we should be able to grant VAT refunds for participants who are registered by businesses, provided we are given all the details we need for VAT exemption.

Each public session will be confirmed once there are at least 6 participants. If the minimum number of participants is not reached, Bootlin will propose new dates or a full refund (including Eventbrite fees) if no new date works for the participant.

We guarantee that the maximum number of participants will be 12.

Dedicated online sessions

If you have enough people to train, such dedicated sessions can be a worthy alternative to public ones:

  • Flexible dates and daily durations, corresponding to the availability of your teams.
  • Confidentiality: freedom to ask questions that are related to your company’s projects and plans.
  • If time left, possibility to have knowledge sharing time with the instructor, that could go beyond the scope of the training course.
  • Language: possibility to have a session in French instead of in English.

Online seminar details

Each session will be given through Zoom. Each participant should have her or his own connection and computer (with webcam and microphone) and if possible headsets, to avoid echo issues between audio input and output. This is probably the best solution to allow each participant to ask questions and write comments in the chat window. We also support people connecting from the same conference room with suitable equipment.

One hour before the session starts, we will organize a test connection, to make sure that everyone has a working set up.

How to register

For online public sessions, use the EventBrite links in the above list of sessions to register one or several individuals.

To register an entire group (for dedicated sessions), please contact training@bootlin.com and tell us the type of session you are interested in. We will then send you a registration form to collect all the details we need to send you a quote.

You can also ask all your questions by calling +33 484 258 097.

Questions and answers

Q: Why just demos instead of practicing with real hardware?
A: Shipping real hardware is going to be complicated to organize in the next weeks, and we are not fully ready to support people doing practical labs remotely with real hardware either. Hence, what we we’re proposing is to replace practical labs with practical demonstrations shown by the instructor. The instructor will go through the normal practical labs with the standard hardware that we’re using.

Q: Why proposing half days instead of full days?
A: From our experience, it’s very difficult to stay focused on a new technical topic for an entire day without having periodic moments when you are active (which happens in our public and on-site sessions, in which we interleave lectures and practical labs). Hence, we believe that daily slots of 4 hours (with 2 or 3 small breaks) is a good solution, also leaving extra time for following up your normal work or taking care of your relatives at home.

Linux 5.5 released, Bootlin contributions inside

Linux 5.5 was recently released, as usual bringing a large number of new features and improvements, which are nicely detailed in the LWN articles on merge window part 1 and merge window part 2, but also on the Kernelnewbies wiki.

According to the statistics, a total of 14350 changes were made to this kernel release, to which Bootlin contributed 124 patches, making us the 19th contributing company by number of commits. Here are the highlights of our contributions:

  • By far and large our most important achievement in Linux 5.5 is the merge of the H265 decoding support in the Allwinner VPU driver, developed by Paul Kocialkowski. This was the last missing feature to complete the effort funded by the Kickstarter campaign we launched in February 2018. See our blog post wrapping up the Allwinner VPU work.
  • Alexandre Belloni as the RTC subsystem maintainer, as usual contributed a large number of RTC driver improvements and fixes.
  • Antoine Ténart contributed some improvements to the Cadence MACB driver, most notably used as the Ethernet controller driver on Microchip/Atmel platforms. The main improvement is the conversion to the phylink subsystem for the interaction with the Ethernet PHY.
  • Grégory Clement contributed numerous enhancements to the Microchip/Atmel SPI controller driver, mainly aimed at fixing and improving the support for Chip Select, both native Chip Selects and GPIO-based Chip Selects.
  • Kamel Bouhara contributed a few additional Device Tree files to describe home automation hardware platforms from Overkiz, based on the Microchip SAMA5D2 processor, as well as an improvement to a Microchip SoC driver that allows to expose the platform serial number to user-space.
  • Miquèl Raynal added support for a new Marvell system-on-chip called Marvell CN9130. Despite the name, this chip is from a hardware point of view in the same family as the Marvell Armada 7K and 8K, which were already supported upstream.
  • Miquèl Raynal contributed a number of fixes and improvements to the Macronix SPI controller driver.
  • Miquèl Raynal added support for the ADC converters over SPI MAX1227, MAX1229 and MAX1231 to the existing max1027 IIO driver in the kernel. These ADCs have a 12-bit resolution.
  • Miquèl Raynal improved the SPI controller driver for the Zynq system-on-chips to correctly support multiple Chip Selects.

In addition to these direct contributions, some of the Bootlin engineers are also Linux kernel maintainers and therefore review and merge patches from other contributors: Alexandre Belloni as the RTC subsystem maintainer and Microchip platform co-maintainer reviewed and merged 45 patches from other contributors, Miquèl Raynal as the MTD subsystem co-maintainer reviewed and merged 39 patches from other contributors and Grégory Clement as the Marvell platform co-maintainer reviewed and merged 29 patches from other contributors.

Here is the detail of our contributions:

Linux 5.4 released, Bootlin contributions inside

LinuxThis time around, we’re quite late to the party, but Linux 5.4 was indeed released a number of weeks ago, and once again, Bootlin contributed a number of patches to this Linux kernel release. As usual, the most useful source of information to learn about the major features brought by Linux 5.4 are the LWN articles (part 1, part 2) and the KernelNewbies Wiki.

With a total of 143 patches contributed to this release, Bootlin is the 17th contributing company by number of commits acccording to the Linux Kernel Patch Statistic.

Here are the highlights of our contributions:

  • Antoine Ténart contributed support for IEEE 1588 Precision Time Protocol (PTP) to the Microsemi Ocelot Ethernet switch driver, which Bootlin developed and upstreamed in 2018 (see our blog post)
  • In the MTD subsystem, a number of contributions to the spi-nor support, written originally by Boris Brezillon, made their way upstream.
  • In the support of Microchip (formerly Atmel) platforms, Kamel Bouhara, who joined Bootlin in September 2019, sees his first kernel contribution merged as a Bootlin engineer: dropping useless support for platform_data from the Atmel PWM driver.
  • In the support of Allwinner platforms
    • Maxime Ripard contributed a brand new driver for the Allwinner A10 camera interface driver, a driver that we started at Bootlin for the CHIP platform back in the days, and that we finished more recently.
    • Maxime Ripard contributed a significant number of improvements to the sun4i-i2s audio interface driver, especially TDM support, which was developed as part of a customer project at Bootlin.
    • Maxime Ripard also contributed numerous enhancements to Allwinner platform Device Tree files, especially in the area of using YAML schemas.
  • In the support for Marvell platforms
    • Grégory Clement added cpufreq support to the Marvell Armada 7K/8K platform by extending some of its clock drivers.
    • Miquèl Raynal contributed improvements to the Marvell CP110 COMPHY driver, which is used to control SERDES lanes on the Marvell Armada 7K/8K platforms, and added the description of the SERDES lanes used by various IP blocks in those processors.
  • Alexandre Belloni, as the RTC subsystem maintainer, did a number of fixes and improvements in several RTC drivers (mainly pcf2123 and pcf8563)
  • For the LPC3250 platform, for which Bootlin delivered a modern BSP to a customer last year, Alexandre Belloni fixed an issue in the lpc_eth network driver, which was preventing the system from booting if the network had been initialized by the bootloader.

In addition to being contributors, some Bootlin engineers are also maintainers of various parts of the Linux kernel, and as such review and merge code from other contributors:

  • As the RTC subsystem maintainer and Microchip platform co-maintainer, Alexandre Belloni merged 47 patches from other contributors
  • As the MTD subsystem co-maintainer, Miquèl Raynal merged 33 patches from other contributors
  • As the Marvell platform co-maintainer, Grégory Clement merged 11 patches from other contributors

Here are the details of all our contributions to Linux 5.4: