Technical – Page 11

ELBE: automated building of Ubuntu images for a Raspberry Pi 3B

Building embedded Linux systems

Typical embedded Linux systems include a wide number of software components, which all need to be compiled and integrated together. Two main approaches are used in the industry to integrate such embedded Linux systems: build systems such as Yocto/OpenEmbedded, Buildroot or OpenWrt, and binary distributions such as Debian, Ubuntu or Fedora. Of course, both options have their own advantages and drawbacks.

One of the benefits of using standard binary distributions such as Debian or Ubuntu is their widespread use, their serious and long-term security maintenance and their large number of packages. However, they often lack appropriate tools to automate the process of creating a complete Linux system image that combines existing binary packages and custom packages.

In this blog post, we introduce ELBE (Embedded Linux Build Environment), which is a build system designed to build Debian distributions and images for the embedded world. While ELBE was initially focused on Debian only, Bootlin contributed support for building Ubuntu images with ELBE, and this blog post will show as an example how to build an Ubuntu image with ELBE for a Raspberry Pi 3B.

ELBE base principle

When you first run ELBE, it creates a Virtual Machine (VM) for building root filesystems. This VM is called initvm. The process of building the root filesystem for your image is to submit and XML file to the initvm, which triggers the building of an image.

The ELBE XML file can contain an archive, which can contain configuration files, and additional software. It uses pre-built software in the form of Debian/Ubuntu packages (.deb). It is also possible to use custom repositories to get special packages into the root filesystem. The resulting root file system (a customized Debian or Ubuntu distribution) can still be upgraded and maintained through Debian’s tools such as APT (Advanced Package Tool). This is the biggest difference between ELBE and other build systems like the Yocto Project and Buildroot.

Bootlin contributions

As mentioned in this blog post introduction, Bootlin contributed support for building Ubuntu images to ELBE, which led to the following upstream commits:

Build an Ubuntu image for the Raspberry Pi 3B

We are now going to illustrate how to use ELBE by showing how to build an image for the popular RaspberryPi 3B platform.

Add required packages

This was tested on Ubuntu 20.04. Install the below packages if needed, and make sure you are in the libvirt, libvirt-qemu and kvm groups:

$ sudo apt install python3 python3-debian python3-mako \
  python3-lxml python3-apt python3-gpg python3-suds \
  python3-libvirt qemu-utils qemu-kvm p7zip-full \
  make libvirt-daemon libvirt-daemon-system \
  libvirt-clients python3-urwid
$ sudo adduser youruser libvirt 
$ sudo adduser youruser libvirt-qemu
$ sudo adduser youruser kvm
$ newgrp libvirt
$ newgrp libvirt-qemu
$ newgrp kvm

Prepare ELBE initvm

First, you need to clone ELBE’s git reposority:

git clone https://github.com/Linutronix/elbe.git

We need to use the v13.2 version because our latest contributions for Ubuntu support made it to 13.2:

$ cd elbe
$ git checkout v13.2

To create the initvm:

$ PATH=$PATH:$(pwd)
$ elbe initvm create --devel

The --devel parameter allows to use ELBE from the current working directory into the initvm.

If the command fails with the Signature with unknown key: message you need to add these keys to apt. Use the following command where XXX is the key to be added:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys XXX

Creating your initvm should take at least 10 to 20 minutes.

In case you rebooted your computer or stopped the VM, you will need to start it:

$ elbe initvm start

Create an ELBE project for our Ubuntu image.

To begin with, we will base our image on the armhf-ubuntu example. We create an ELBE pbuilder project and not a simple ELBE project because we later want to build our own Linux kernel package for our board:

$ elbe pbuilder create --xmlfile=examples/armhf-ubuntu.xml \
  --writeproject rpi.prj --cross

The project identifier is written to rpi.prj. We save the identifier to a shell variable to simplify the next ELBE commands:

$ PRJ=$(cat rpi.prj)

Build the Linux package

As explained earlier we want to use ELBE to build our package for the Linux kernel. ELBE uses the standard Debian tool pbuilder to build packages. Therefore, we need to have debianized sources (i.e sources with the appropriate Debian metadata in a debian/ subfolder) to build a package with pbuilder.

First clone the Linux repositories:

$ git clone -b rpi-5.10.y https://github.com/raspberrypi/linux.git
$ cd linux

Debianize the Linux repositories. We use the elbe debianize command to simplify the generation of the debian folder:

$ elbe debianize

Fill the settings in the UI as follows (make sure you reduce the font size if you don’t see the Confirm button):

Make sure you set Name to rpi. Otherwise, you won’t get the output file names we use in the upcoming instructions.

The debianize command helps to create the skeleton of the debian folder in the sources. It has been pre-configured for a few packages like bootloaders or the Linux kernel, to create the rules to build these packages. It may need further modifications to finish the packaging process. Take a look a the manual to have more information on debianization. In our case, we need to tweak the debian/ folder with the two following steps to cross-build the Raspberry Linux kernel without error.

Append the below lines to the debian/rules file (use tabs instead of spaces):

override_dh_strip:
	dh_strip -Xscripts

override_dh_shlibdeps:
	dh_shlibdeps -Xscripts

Remove the following line from the debian/linux-image-5.10-rpi.install file:

./lib/firmware/*

Update the source format:

$ echo "1.0" > debian/source/format

The Linux kernel sources are now ready, we can run elbe pbuilder to compile them:

$ mkdir ../out
$ elbe pbuilder build --project $PRJ --cross --out ../out

According to how fast your system is, this can run for hours!

If everything ends well without error the out/ directory has been filled with output files:

$ ls ../out
linux-5.10-rpi_1.0_armhf.buildinfo
linux-5.10-rpi_1.0_armhf.changes
linux-5.10-rpi_1.0.dsc
linux-5.10-rpi_1.0.tar.gz
linux-headers-5.10-rpi_1.0_armhf.deb
linux-image-5.10-rpi_1.0_armhf.deb
linux-libc-dev-5.10-rpi_1.0_armhf.deb

Update the Ubuntu XML image description

Now we have our Linux kernel packaged we can move on to the image generation. Since we started from examples/armhf-ubuntu.xml, we will modify this file to fit our needs.

We begin by adding the Linux kernel package to the XML image description in the pkg-list node:


<pkg-list>
...
	<pkg>linux-image-5.10-rpi</pkg>
...
</pkg-list>

We also have to add the Device Tree to the boot/ directory because the Linux kernel package installs all the Device Trees into the /usr/lib directory.

This change is part of the rootfs modifications, therefore it is described under the finetuning XML node. We also rename the kernel image to kernel.img:


<finetuning>
...
	<cp path="/usr/lib/linux-image-5.10-rpi/bcm2710-rpi-3-b.dtb">/boot/bcm2710-rpi-3-b.dtb</cp>
	<cp path="/usr/lib/linux-image-5.10-rpi/overlays">/boot/overlays</cp>
	<mv path="/boot/vmlinuz-5.10-rpi">/boot/kernel.img</mv>
...
</finetuning>

We want to use an SD card on our Raspberry Pi, so we have to describe the partitioning of our image. For this purpose, we add the images and the fstab XML nodes to the target XML node:


<target>
...
	<images>
		<msdoshd>
			<name>sdcard.img</name>
			<size>1500MiB</size>
				<partition>
					<size>50MiB</size>
					<label>boot</label>
					<bootable/>
				</partition>
				<partition>
					<size>remain</size>
					<label>rfs</label>
				</partition>
		</msdoshd>
	</images>
	<fstab>
		<bylabel>
			<label>rfs</label>
			<mountpoint>/</mountpoint>
			<fs>
				<type>ext2</type>
			</fs>
		</bylabel>
		<bylabel>
			<label>boot</label>
			<mountpoint>/boot</mountpoint>
			<fs>
				<type>vfat</type>
			</fs>
		</bylabel>
	</fstab>
...
</target>

The Raspberry Pi board also needs firmware binaries and configurations file to boot properly. We will use the overlay directory to add these Raspberry firmware files to the image:

$ mkdir -p overlay/boot
$ cd overlay/boot
$ wget https://github.com/raspberrypi/firmware/raw/1.20210201/boot/bootcode.bin
$ wget https://github.com/raspberrypi/firmware/raw/1.20210201/boot/start.elf
$ wget https://github.com/raspberrypi/firmware/raw/1.20210201/boot/fixup.dat
$ echo "console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootwait" > cmdline.txt
$ echo "dtoverlay=miniuart-bt" > config.txt

ELBE stores the overlay uuencoded in the XML file using the chg_archive command:

$ elbe chg_archive examples/armhf-ubuntu.xml overlay

The archive node got created in the XML file.

To tell ELBE that the XML file has changed, you need to send it to the initvm:

$ elbe control set_xml $PRJ examples/armhf-ubuntu.xml

Then build the image with ELBE:

$ elbe control build $PRJ
$ elbe control wait_busy $PRJ

Finally, if the build completes successfully, you can retrieve the image file from the initvm:

$ elbe control get_files $PRJ
$ elbe control get_file $PRJ sdcard.img.tar.gz

Now you can flash the SD card image:

$ tar xf sdcard.img.tar.gz
$ dd if=sdcard.img of=/dev/sdX bs=1M

And boot the board with root and foo as login and password:

Ubuntu 18.04.1 LTS myUbuntu ttyAMA0

myUbuntu login: root
Password: 
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 5.10-rpi armv7l)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@myUbuntu:~#

Note: Ubuntu cannot be built for Raspberry A, B, B+, 0 and 0W according to https://wiki.ubuntu.com/ARM/RaspberryPi, as Ubuntu targets the ARMv7-A architecture, while the older RaspberryPi use an ARMv6 processor.

Further details

You can find our talk about ELBE at the ELCE 2020 conference: PDF slides, video.
See also ELBE’s official documentation

LTP: Linux Test Project, Bootlin contributions

Introduction

The Linux Test Project is a project that develops and maintains a large test suite that helps validating the reliability, robustness and stability of the Linux kernel and related features. LTP has been mainly developed by companies such as IBM, Cisco, Fujitsu, SUSE, RedHat, with a focus on desktop distributions.

On the embedded side, both the openembedded-core Yocto layer and Buildroot have packages that allow to use LTP on embedded targets. However, for a recent project, we practically tried to run the full LTP test suite on an i.MX8 based platform running a Linux system built with Yocto. It turned out that LTP was apparently not very often tested on Busybox-based embedded systems, and we faced a number of issues. In addition to reporting various bugs/issues to the upstream LTP project, we also contributed a number of fixes and improvements:

Our contributions received a very warm welcome in the LTP community, which turned out to be very open and responsive. We hope that these contributions will encourage others to use LTP, and hopefully to make sure it continues to work on embedded platforms.

Quick start guide

At the time of this writing, LTP has more than 3800 tests written by the community, including about 1000 network-related tests. The tests are grouped together in categories described by files in the runtest/ folder. Based on this, two scenarios of tests are defined: default and network which are described by two files in the scenario_groups/ folder. These two scenarios simply list the categories of tests that need to be executed.

Here are the contents of the default and network:

$ cat scenario_groups/default 
syscalls
fs
fs_perms_simple
fsx
dio
io
mm
ipc
sched
math
nptl
pty
containers
fs_bind
controllers
filecaps
cap_bounds
fcntl-locktests
connectors
power_management_tests
hugetlb
commands
hyperthreading
can
cpuhotplug
net.ipv6_lib
input
cve
crypto
kernel_misc
uevent

$ cat scenario_groups/network 
can
net.features
net.ipv6
net.ipv6_lib
net.tcp_cmds
net.multicast
net.rpc
net.nfs
net.rpc_tests
net.tirpc_tests
net.sctp
net_stress.appl
net_stress.broken_ip
net_stress.interface
net_stress.ipsec_dccp
net_stress.ipsec_icmp
net_stress.ipsec_sctp
net_stress.ipsec_tcp
net_stress.ipsec_udp
net_stress.multicast
net_stress.route

Once you have LTP built and installed on your board thanks to the appropriate OpenEmbedded or Buildroot package, you can run these two scenarios of test with the following commands (-n specify the network one):

$ cd /opt/ltp
$ ./runltp
$ ./runltp -n

Then take a look at the content of the result and the output directories.

For more information on building or running LTP please read this readme.

Bootlin welcomes Hervé Codina in its team

Since March 1st, 2021, we’re happy to have an additional engineer, Hervé Codina, in our engineering team based in Toulouse, France.

Hervé has 20 years’ experience working in embedded systems, both bare-metal systems and embedded Linux systems, in a wide range of applications. Hervé has experience working with U-Boot, Barebox, Linux, Buildroot, Yocto, on ARM platforms from various silicon vendors. Hervé will work within our engineering team to deliver ready-to-use Linux Board Support Packages, port bootloaders and the Linux kernel to new platforms, develop Linux kernel device drivers, implement custom Linux systems with Buildroot or Yocto, and more. His 20 years experience will further increase the expertise that Bootlin provides to its worldwide customers.

See Hervé’s page on our site for more details. Hervé is joining our team in Toulouse, who already included Paul Kocialkowski, Miquèl Raynal, Köry Maincent, Maxime Chevallier, Thomas Perrot and Thomas Petazzoni.

Bootlin contributions to Linux 5.11

Linux 5.11 was released quite some time ago now, but it’s never too late to have a look at Bootlin contributions to this release. As usual, we recommend reading the LWN articles on the 5.11 merge window: part 1 and part2. Also of interest is the Kernelnewbies page for 5.11.

Here are the main highlights of our contributions:

Alexandre Belloni, as the maintainer of the RTC subsystem, continued making numerous improvements and fixes to RTC drivers
On the support for Microchip ARM platforms, Alexandre Belloni switched the PWM atmel-tcb driver to a new Device Tree binding and added SAMA5D2 support, he did some improvements to the IIO driver for the Microchip ADC, and continued to remove platform_data support from Microchip drivers as all platforms are now converted to the Device Tree.
Alexandre Belloni contributed a new Simple Audio Mux driver for the ALSA subsystem, which can be used to control simple audio multiplexers driven using GPIOs, that allows to select which of their input line is connected to the output line.
Grégory Clement added support for several new MIPS platforms from Microchip: Luton, Serval and Jaguar2. All those platforms include a MIPS core, a few peripherals and more importantly an Ethernet switch. For now the support only includes the base platform support, but we are working on the switchdev driver for the Ethernet switch.
Miquèl Raynal, maintainer of the NAND subsystem and co-maintainer of the MTD subsystem, contributed numerous changes to the ECC support in the MTD subsystem, making it more generic so that it can be used not just for parallel NAND flashes, but also SPI NAND flashes. For more details, see the talk from Miquèl Raynal on this topic.

In addition to those 95 patches that we authored and contributed, several Bootlin engineers being maintainers of different subsystems of the Linux kernel reviewed and merged patches from other contributors:

Miquèl Raynal, as the NAND maintainer and MTD co-maintainer, reviewed and merged 67 patches from other contributors
Alexandre Belloni, as the RTC, I3C and Microchip ARM/MIPS platforms maintainer, reviewed and merged 47 patches from other contributors
Grégory Clement, as the Marvell EBU platform co-maintainer, reviewed and merged 33 patches from other contributors

Here is the detailed list of our contributions to Linux 5.11:

Videos and slides of Bootlin presentations at FOSDEM 2021

The videos from Bootlin’s presentations earlier this month at FOSDEM 2021 are now publicly available. Once again, FOSDEM was a busy event, even if it was online for once. As in most technical conferences, Bootlin engineers volunteered to share their experience and research by giving two talks.

Maxime Chevallier – Network Performance in the Linux Kernel, Getting the most out of the Hardware

Abstract: The networking stack is one of the most complex and optimized subsystems in the Linux kernel, and for a good reason. Between the wild range of applications, the complexity and variety of the networking hardware, getting good performance while keeping the stack easily usable from userspace has been a long-standing challenge.

Nowadays, complex Network Interface Controllers (NICs) can be found even on small embedded systems, bringing powerful features that were previously found only in the server world closest to day to day users.

This is a good opportunity to dive into the Linux Networking stack, to discover what is used to make networking as fast as possible, both by using all the features of the hardware and by implementing some clever software tricks.

In this talk, we cover these various techniques, ranging from simple batch processing with NAPI, queue management with RSS, RPS, XPS and so on, flow steering and filtering with ethool and TC, to finish with the newest big change that is XDP.

We dive into these various techniques and see how to configure them to squeeze the most out of your hardware, and discover that what was previously in the realm of datacenters and huge computers can now also be applied to embedded linux development.

Here are PDF slides for this presentation.

Michael Opdenacker – Embedded Linux from Scratch in 45 minutes, on RISC-V

Abstract: Discover how to build your own embedded Linux system completely from scratch. In this presentation and tutorial, we show how to build a custom toolchain (Buildroot), bootloader (opensbi / U-Boot) and kernel (Linux), that one can run on a system with the new RISV-V open Instruction Set Architecture emulated by QEMU. We also show how one can build a minimal root filesystem by oneself thanks to the BusyBox project. The presentation ends by showing how to control the system remotely through a tiny webserver. The approach is to provide only the files that are strictly necessary. That’s all the interest of embedded Linux: one can really control and understand everything that runs on the system, and see how simple the system can be. That’s much easier than trying to understand how a GNU/Linux system works from a distribution as complex as Debian!

The presentation also shares details about what’s specific to the RISC-V architecture, in particular about the various stages of the boot process. This presentation shares all the hardware (!), source code build instructions and demo binaries needed to reproduce everything at home, and add specific improvements. Most of the details are also useful to people using other hardware architectures (in particular arm and arm64).

It’s probably the first time a tutorial manages to show so many aspects of embedded Linux in less than an hour. See by yourself! At least, that’s for sure the first one demonstrating how to boot Linux from U-Boot in a RISC-V system emulated by QEMU.

Here are PDF slides for this presentation.

Device Tree 101 webinar slides and videos

As we announced back in January, we have offered in partnership with ST on February 9 a free webinar titled Device Tree 101, which gives a detailed introduction to the Device Tree, an important mechanism used in the embedded Linux ecosystem to describe hardware platforms. We were happy to see the interest around this topic and webinar.

Bootlin has always shared freely and openly all its technical contents, including our training materials, and this webinar is no exception. We are therefore sharing the slides and video recording of both sessions of this webinar:

Thanks to everyone who participated and thanks to ST for the support in organizing this event! Do not hesitate to share and/or like our video, and to suggest us other topics that would be useful to cover in future webinars!

Free “Device Tree 101” webinar, on February 9, 2021

In partnership with ST, we are organizing on February 9, 2021, a free webinar entitled “Device Tree 101”.

The Device Tree has been adopted for the ARM 32-bit Linux kernel support almost a decade ago, and since then, its usage has expanded to many other CPU architectures in Linux, as well as bootloaders such as U-Boot or Barebox. Even though Device Tree is no longer a new mechanism, developers coming into the embedded Linux world often struggle to understand what Device Trees are, what is their syntax, how they interact with the Linux kernel device drivers, what Device Tree bindings are, and more. This webinar will offer a deep dive into the Device Tree, to jump start new developers in using this description language that is now ubiquitous in the vast majority of embedded Linux projects. This webinar will be illustrated with numerous examples applicable to the STM32MP1 MPU platforms, which make extensive usage of the Device Tree.

This webinar will take place on February 9, 2021, and is proposed at two different times during the day: at 10 AM CET (UTC+1) and 5 PM CET (UTC+1). The duration of the webinar is 1 hours and 30 minutes. Registration is free at https://www.eventbrite.com/e/135964923747. The webinar itself will be hosted as a Youtube Live stream, which will allow participants to ask questions in the chat during the webinar.

The trainer for this webinar is Thomas Petazzoni, Bootlin’s CTO. Thomas is the author of the popular « Device Tree for Dummies » talk given in 2014 and which helped numerous embedded Linux developers get started with the Device Tree. Thomas has contributed over 900 patches to the official Linux kernel, mainly around ARM hardware platform support. He is also the co-maintainer of the Buildroot open-source project.

Bootlin toolchains integration in Buildroot

Since 2017, Bootlin is freely providing ready-to-use pre-built cross-compilation toolchains at https://toolchains.bootlin.com/. We are now providing over 150 toolchains, for a wide range of CPU architectures, covering the glibc, uClibc-ng and musl C libraries, with up-to-date gcc, binutils, gdb and C library support.

We recently contributed an improvement to Buildroot that allows those toolchains to very easily be used in Buildroot configurations: the Bootlin toolchains are now all known by Buildroot as existing external toolchains, next to toolchains from other vendors such as ARM, Synopsys and others.

If you are building a Buildroot system for a CPU architecture variant that has a matching toolchain available from bootlin.toolchains.com, then Bootlin toolchains will naturally show up in the Toolchain sub-menu, when the selected Toolchain type is External toolchain. For example, if the selected CPU architecture is ARM little endian Cortex-A9, with VFP you will see:

Once Bootlin toolchains is selected, a new sub-option Bootlin toolchain variant appears, which allows to choose between the different toolchains applicable to the selected CPU architecture:

This hopefully should make Bootlin toolchains easier to use for Buildroot users.

Internally, this support for Bootlin toolchains in Buildroot is generated and updated using the support/scripts/gen-bootlin-toolchains script. In addition to making the toolchains available to the user, it allows generates some Buildroot test cases for each toolchain, so that each of those configuration is tested by Buildroot continuous integration, see support/testing/tests/toolchain/test_external_bootlin.py.

Linux 5.10, Bootlin contributions

Linux 5.10 was released a few weeks ago, and while 5.11-rc2 is already out, it’s still time to look at what Bootlin contributed to the 5.10 kernel. As usual, for a broad overview of the major changes in 5.10, we recommend reading the LWN articles: 5.10 merge window part 1, the rest of the 5.10 merge window, or the 5.10 KernelNewbies page.

Overall, Bootlin contributed 78 patches to this kernel release, in the following areas:

Alexandre Belloni did a number of improvements in the support of Microchip ARM platforms: device tree updates, code cleanups, etc.
Alexandre Belloni added a new rv3032 RTC driver and did some improvements to the r9701 RTC driver.
Miquèl Raynal implemented a significant rework of how ECC engines are handled in the MTD subsystems, so that ECC engines can be used not just for parallel NANDs but also for SPI NANDs. See also the talk that Miquèl gave at the Embedded Linux Conference Europe on this topic: slides and video.
Miquèl Raynal contributed a few improvements to the tlv320aic32x4 audio codec driver.
Paul Kocialkowski made some small improvements, one in the OV5640 camera sensor driver, and one in the Rockchip DRM driver.
Thomas Petazzoni implemented a performance improvement in the max310x driver, used for SPI-connected UART controllers.

In addition to these code contributions, we also contribute by having several of our engineers be maintainers of a few subsystems of the Linux kernel. As part of this:

Miquèl Raynal reviewed and merged 47 patches touching the MTD subsystem he co-maintains with other kernel developers.
Alexandre Belloni reviewed and merged 42 patches touching either the Microchip ARM or MIPS platforms, or the RTC subsystem.
Grégory Clement reviewed and merged 2 patches touching the Marvell ARM/ARM64 platform support.

Here is the complete list of our commits to 5.10.

Bootlin welcomes Thomas Perrot in its team

Since December 1st, 2020, we’re happy to have in our team an additional engineer, Thomas Perrot, who joined our office in Toulouse, France.

Thomas brings 6+ years of experience working on embedded Linux systems, during which he worked at Intel on Android platforms, and then at Sigfox on the base stations for Sigfox’s radio network. Thomas has experience working with Linux on x86-64, ARM and ARM64 platforms, with a wide range of skills: bootloader development, Linux kernel and driver development, Yocto integration, OTA updates. Thomas was also deeply involved in the strong security aspects of Sigfox base stations, with secure boot and measured boot, TPM, integrity measurement, etc. At Sigfox, Thomas was involved in all steps of the product life-cycle, from the design phase all the way to the in-field deployment, update and maintenance. Last but not least, Thomas is a Linux technologist and a free software enthusiast, who hacks some open source hardware projects on his free time. See also Thomas page on our website and his LinkedIn profile.

Thomas Perrot is joining our growing team of engineers in Toulouse, which already included Paul Kocialkowski, Miquèl Raynal, Köry Maincent, Maxime Chevallier and Thomas Petazzoni.