Bootlin has been for many years a key contributor to the Buildroot project, a very popular embedded Linux build system. A few years ago, we decided to share our Buildroot expertise by creating a corresponding training course: Embedded Linux development with Buildroot, for which the training materials are freely available, under a Creative Commons license.

We have recently updated this training course up to Buildroot 2019.02, which is the latest “long term support” release of the project. Both the lectures and practical labs have been updated to this Buildroot version.

In addition, the board used in the course has been changed to the BeagleBone Black Wireless, instead of the BeagleBone Black, which is no longer easily available. The practical labs were updated accordingly, and we now use the USB device interface to provide network connectivity between the development PC and the embedded target.

This 3-day Buildroot training course can be delivered on-site at your location, anywhere in the world. See our cost and registration page for more details.

Today, the three Raspberry Pis that we have on our network went down. They were all running Raspbian (Debian for Raspberry Pi) Stretch.

While this issue can be solved, it is serious enough to require to remove the micro-SD card and manually fix the the root filesystem. Therefore, it seems you cannot fix this issue unless you have physical access to your system.

Here are details to attract attention to this issue…

As I started telling you, our systems were down, well almost. Some services were still running, as they were still responding to ping through our VPN. However, SSH access was no longer available:

$ ssh scan
ssh_exchange_identification: Connection closed by remote host

After connecting a serial cable to one of the Pis, and adding init=/bin/sh to the /boot/cmdline.txt file. I found that I couldn’t seem to execute at least some executables. Everything I tried to execute was causing a segmentation fault.

It was time to remove the micro-SD card and look at system logs. Inspecting /var/log/apt/history.log revealed that the raspi-copies-and-fills package was updated yesterday (March. 11, 2019). This allowed me to make a search for issue reports with this package name. Indeed, before having such a lead, I couldn’t find what I was looking for, as there are too many discussions about the use of the Raspberry Pi! So, here’s what I quickly found following this lead:

raspi-copies-and-fills package gone AWOL? (Raspberry Pi forums)
Ras3 crashes after update (Raspberry Pi forums)

These posts have all details. All you need to do is take away the micro-SD card, repair the second partition with e2fsck -f /dev/mmcblk0p2 and remove the etc/ld.so.preload file inside this partition.

Note, that at the time of this writing, this issue has already been fixed, so it is safe to upgrade your Pi if it is still up and running, or right after repairing your Raspbian root filesystem.

This incident is very unfortunate, as you need to physically access your board to recover from it. I hope you don’t run updates as frequently as we do (or right after the time when the update was issued), and that your Pis are not impacted, otherwise possibly forcing you to travel or to crawl into difficult to access places to reach your boards.

However, I don’t want blame the community volunteers running Raspbian. They have made a terrific job maintaining this distro which had been flawlessly running for more many years on our systems. This seems as good as what we get from commercial distributions.

I hope that not too many services ran by Raspberry Pis will be disrupted because of this issue. However, that may be yet another way to prove how popular such devices are.

MACsec is an IEEE standard (IEEE 802.1AE) for MAC security, introduced in 2006. It defines a way to establish a protocol independent connection between two hosts with data confidentiality, authenticity and/or integrity, using GCM-AES-128. MACsec operates on the Ethernet layer and as such is a layer 2 protocol, which means it’s designed to secure traffic within a layer 2 network, including DHCP or ARP requests. It does not compete with other security solutions such as IPsec (layer 3) or TLS (layer 4), as all those solutions are used for their own specific use cases.

We have recently worked on enabling hardware offloading of MACsec operations on a Microsemi VSC8584 Ethernet PHY in Linux, by contributing support for MACsec offloading to the Linux networking stack. In this blog post, we present this work through an introduction to MACsec, details on the current state of MACsec support in Linux and finally our work to support MACsec hardware offloading.

Introduction to MACsec

MACsec uses its own frame format with its own EtherType (a 2-bytes field found in Ethernet frames to indicate what the protocol encapsulated in the payload is). As an example, when encapsulating an IPv4 frame, we would have Ethernet<MACsec<IPv4 instead of Ethernet<IPv4.

The MACsec configuration within a node is represented at the top level by Secure Channels. A secure channel is identified by its SCI (Secure Channel Identifier) and contains parameters such as the encryption, protection and replay protection booleans. A secure channel is either a transmit or a receive one: the receive secure channel configuration on a given host should match the transmit one of another host for MACsec traffic to flow successfully.

Within each secure channel, security associations are described. They are identified by an association number and define the encryption/decryption keys used and the current packet number, which is used for replay protection.

MACsec support in Linux

Linux has a software implementation of MACsec, found at drivers/net/macsec.c, which was introduced by Red Hat engineer Sabrina Dubroca in 2016 and available since Linux 4.5. The support is implemented as full virtual network devices, on per transmit secure channel, attached to a parent network device. The parent interface only sees raw packets, which are in the MACsec case raw Ethernet packets with protected or encrypted content. This design is very similar to other supported protocols in Linux such as VLANs.

MACsec support was also introduced in iproute2, a collection of utilities aiming at configuring various networking parts of the kernel (interfaces management, IP configuration, routes…). The command to use is ip macsec.

If we were to configure a secure channel between two hosts we would first need to create a virtual MACsec interface (representing a transmit secure channel) on both hosts, on top of a physical network interface. Let’s say we use eth0 on both our hosts (Alice and Bob), and we want to encrypt the MACsec traffic:

Alice # ip macsec add link eth0 macsec0 type macsec encrypt on
  Bob # ip macsec add link eth0 macsec0 type macsec encrypt on

The next step would be to configure matching receiving secure channel on both hosts:

Alice # ip macsec add macsec0 rx port 1 address <Bob's eth0 MAC>
  Bob # ip macsec add macsec0 rx port 1 address <Alice's eth0 MAC>

We would then configure the transmit channels, and for each we would need to generate a key:

Alice # hexdump -n 16 -e '4/4 "%08x" 1 "\n"' /dev/random
d29a43c8cba96a325f6b6a40a214c58c
Alice # ip macsec add macsec0 tx sa 0 pn 1 \
        on key d29a43c8cba96a325f6b6a40a214c58c

Bob # hexdump -n 16 -e '4/4 "%08x" 1 "\n"' /dev/random
a1e15a1d91222196fde87b2d75a4fac0
Bob # ip macsec add macsec0 tx sa 0 pn 1 \
      on key a1e15a1d91222196fde87b2d75a4fac0

We finally need to configure the receive channels, so that the hosts can authenticate and decrypt packets:

Alice # ip macsec add macsec0 rx port 1 \
        address <Bob's MAC> sa 0 pn 1   \
        on key 00 a1e15a1d91222196fde87b2d75a4fac0
 
  Bob # ip macsec add macsec0 rx port 1 \
        address <Alice's MAC> sa 0 pn 1 \
        on key 00 d29a43c8cba96a325f6b6a40a214c58c

Once all of the MACsec configuration is done we would be able to exchange traffic between Alice and Bob, using authenticated and encrypted packets:

Alice # ip link set macsec0 up
Alice # ip addr add 192.168.42.1/24 dev macsec0

  Bob # ip link set macsec0 up
  Bob # ip addr add 192.168.42.2/24 dev macsec0

What’s coming next: hardware offloading

There are hardware devices featuring a MACsec transformation implementation which can be used to offload the frame generation and encryption / authentication of MACsec frames (for both ingress and egress frames). The benefit of hardware offloading is to discharge the CPU from doing certain operations (in our case MACsec transformations) by doing them in a dedicated hardware engine, which may or may not provide better performance. The idea is essentially to free the CPU from being used by a single application so that the system in its whole runs better.

MACsec offloading devices aren’t currently supported in the Linux kernel and no generic infrastructure is available to delegate MACsec operation to a given hardware device. At Bootlin over the last months we worked on adding such an infrastructure and support for offloading MACsec operations to a first device.

This work was done in two steps. First we needed to extend the current MACsec implementation to propagate commands and configuration to hardware drivers. Our idea was to leverage the current MACsec software implementation to use the exact same commands described above to setup an hardware accelerated MACsec connection, when a Linux networking port supports it. This should allow to have a more maintainable implementation as well.

We then worked on implementing a MACsec specific helper in a networking PHY driver : the Microsemi VSC8584 Ethernet PHY. This PHY has a MACsec engine which can be used to match flows and to perform MACsec transformations and operations. When configured packets can be encrypted and decrypted, protected and validated, without the CPU intervention.

Conclusion

We recently sent a first version of patch series to the Linux network mailing list, which is currently being discussed. This series of patches introduces both the hardware offloading support for MACsec and the ability to offload MACsec operations to a first hardware engine. We hope support for other MACsec engines will come after!