Continuous integration in Yocto: improving the regressions detection

The Yocto Project is an open source umbrella project which gathers all needed tools to build full Linux distributions for a wide variety of devices. As the interest for Yocto grew since its first steps, its size and number of use cases increased consequentially. This growth quickly introduced the need of automated testing so that developers can keep introducing new features to the project while making sure not to break any existing part. Bootlin engineer Alexis Lothoré has recently been involved in the Continuous Integration infrastructure of the Yocto Project and has brought improvements to allow Yocto maintainers to detect regressions earlier.

Continue reading “Continuous integration in Yocto: improving the regressions detection”

Embedded Linux and Yocto training courses now available in Italian

Flag of ItalyBuongiorno!

For many years, we have been offering our training courses in both English and French. We are happy to announce that we are now expanding our offering towards our Italian customers: we are now able to deliver our very popular Embedded Linux system development and Yocto Project and OpenEmbedded development training courses in Italian. Those courses are delivered by a native Italian speaker, Bootlin engineer Luca Ceresoli. Luca has 20 years of experience working with embedded systems, including 15 years on embedded Linux.

Thanks to this offering, our Italian customers can enjoy a training course delivered in their native language, which obviously facilitates communication, understanding and interaction. We can offer our Embedded Linux course and Yocto course in Italian through either private on-site sessions or private on-line sessions. Contact us at training@bootlin.com for details and pricing!

Ti aspettiamo!

Updated Yocto support for SiFive RISC-V platforms

SiFive logoA few months ago, we started supporting SiFive in their effort to maintain and improve the Yocto support for their RISC-V processors and platforms. The Yocto support for SiFive platforms is divided into two layers:

These layers allow to build ready-to-use Yocto images for the Qemu RISC-V 64-bit emulation, the HiFive Unleashed and HiFive Unmatched development boards.

As part of our work on the Yocto support for SiFive platofrms, we have already published three new releases of these layers: 2022.06, 2022.08 and 2022.10. In this blog post, we review the main highlights of those releases.

Continue reading “Updated Yocto support for SiFive RISC-V platforms”

Upcoming Yocto Project Summit 2022.11 – One talk from Bootlin

Headline of the Yocto Project Summit 2022.11.

As every six months for the last two years, a new virtual edition of the Yocto Project Summit is coming, and its schedule has been announced.

This summit will be over 3 days:

  • Tuesday, November 29
    Two tracks in parallel, a beginner track and a “hands-on” track for people already familiar with the concepts.
  • Wednesday, November 30
    Only one track, with intermediate level talks on all kinds of topics.
  • Thursday, December 1
    Only one track, starting with “product showcase” talks and going on with intermediate level talks on various topics too.

Last but not least, at the end of each day, you will get a chance to hangout with other contributors and users, and ask all the questions that you may have.

Bootlin is proud to contribute one talk to this summit: Bitbaking SPDX SBoM which we will prepare and present. This talk will cover one of the topics we explored to document features of the Yocto Project which had no documentation yet. SPDX and SBoM are related to software supply chain security, vulnerability management and license compliance. These are hot topics these days, and there will be another presentation about SBoMs (SBoMs and Supply Chain with the Yocto Project by Joshua Watt), and two about security (Detecting and fixing CVE security issues in yocto based embedded Linux distribution by Mikko Rapeli, and Maintenance and Security of a Yocto Project-based Distribution: A Year of Experiences by Marta Rybczynska).

Bootlin is indeed involved in the Yocto Project by maintaining its documentation (see the active contributors through the git repository), and also a participant to the Yocto SWAT team, keeping track of all the issues encountered by the autobuilder machines and runs.

Though the Yocto Project and OpenEmbedded are Open Source projects, registration for this conference is not free but just costs 40 USD, to cover infrastructure and staffing costs, the event being hosted by the Linux Foundation.

Yocto training course updated to Kirkstone release

Yocto ProjectBack in May 2022, the Yocto Project published the Kirkstone release, the latest Long Term Support version of the popular embedded Linux build system. See the release notes of this 4.0 release.

For many years, Bootlin has helped plenty of engineers around the world get started with the Yocto Project and OpenEmbedded thanks to our Yocto Project and OpenEmbedded development training course, whose training materials are freely accessible, to everyone.

We are happy to announce that we have just published the update of these materials to cover this new Kirstone release. Our practical lab instructions, available for both the STM32MP1 Discovery Kit and the BeagleBoneBlack, have been correspondingly updated:

This update was done by Bootlin engineer Luca Ceresoli, who is teaching this updated version of our course this week. The next public session, open to individual registration, will take place on September 26-30, and will be taught by Maxime Chevallier. If you are interested, you can register directly online. We also offer this course in private sessions, organized on demand for your team, either on-line or on-site, you can contact us for more details.

Using Flutter on NVidia Jetson to build graphical applications

Introduction

Flutter is an open source UI framework, released in 2017 by Google, that allows the creation of multi-platform applications, without having to worry about constraints related to supported platforms.

Flutter applications are written in a programming language called Dart, then compiled and run as a native applications, to be efficiently executed on Linux, Android, iOS or Windows platforms, but also as Web applications.

On Linux platforms, Flutter can be used on top of several graphic back-ends:

  • DRM
  • Wayland
  • X11

Flutter is composed of four main parts:

  • the embedder (C++, Java…): the glue for specific platforms that provides surface rendering, vsync.
  • the engine (C/C++): the graphic engine based on Skia, that provides asset resolution, graphics shell, Dart VM…
  • the framework (Dart): to create UI by using widgets, animation…
  • the applications (Dart)

In this article, we show how to build a custom Linux distribution that includes a Flutter Embedder that uses the DRM/EGLStream backend, in order to run the Flutter Gallery application on the NVIDIA Tegra Xavier NX platform.

In addition, we will also extend a Yocto SDK to embed the Flutter toolchain, to be able to build Flutter applications directly with the SDK.

Configure Yocto and build an image

To build our Flutter-enabled Linux distribution, we have chosen to use OpenEmbedded, driven through the Kas utility. Kas is a tool developed by Siemens to facilitate the setup of projects based on Bitbake, such as OpenEmbedded or Yocto.

Kas relies on a YAML file that indicates the information required to:

  • clone bitbake and required layers
  • configure the build environment
  • launch bitbake process

Below is the Kas YAML file that we created for this example:

header:
  version: 11

build_system: oe

machine: jetson-xavier-nx-devkit

distro: nodistro

target:
 - core-image-minimal

repos:

  bitbake:
    url: https://git.openembedded.org/bitbake
    refspec: "2.0"
    layers:
      .: excluded

  openembedded-core:
    url: https://git.openembedded.org/openembedded-core
    refspec: kirkstone
    layers:
      meta:

  meta-clang:
    url: https://github.com/kraj/meta-clang.git
    refspec: kirkstone
    layers:
      .:

  meta-flutter:
    url: https://github.com/meta-flutter/meta-flutter.git
    refspec: kirkstone
    layers:
      .:

  meta-tegra:
    url: https://github.com/OE4T/meta-tegra.git
    refspec: kirkstone
    layers:
      .:

local_conf_header:
  standard: |
    FLUTTER_RUNTIME = "release"
    FLUTTER_SDK_TAG = "3.0.1"

    DISTRO_FEATURES:append = " opengl wayland"
    REQUIRED_DISTRO_FEATURES:append = " opengl wayland"

    IMAGE_INSTALL:append = " flutter-drm-eglstream-backend flutter-gallery-release tegra-udrm-probeconf"

    TOOLCHAIN_HOST_TASK:append = " nativesdk-flutter-sdk"
    TOOLCHAIN_TARGET_TASK:append = " gtk+3-dev"

    CLANGSDK = "1"
    INIT_MANAGER ?= "systemd"

Several OpenEmbedded layers are used:

  • Obviously the openembedded-core layer, the base
  • The meta-flutter layer, which contains all the Flutter related recipes
  • The meta-tegra layer, which is our BSP layer containing all the bootloader/kernel and machine-specific recipes for the Nvidia Jetson Xavier NX platform
  • The meta-clang layer, which is needed by the meta-flutter layer, as Flutter is built using the Clang compiler

As we chose to use a DRM/EGLStream backend of Flutter we extend the DISTRO_FEATURES to enable the support of OpenGL and Wayland:

DISTRO_FEATURES:append = " opengl wayland"

In addition, we extend the list of packages that will be installed in the image with the ones that provide the Flutter embedder and the Gallery application:

IMAGE_INSTALL:append = " flutter-drm-eglstream-backend flutter-gallery-release"

Moreover, as we want to use the release 2.10.5 of Flutter, and without debug support:

FLUTTER_RUNTIME = "release"
FLUTTER_SDK_TAG = "3.0.1"

Finally, we also install the package that provides a configuration to set the correct modeset when the Tegra direct rendering module is probed.

IMAGE_INSTALL:append = " tegra-udrm-probeconf"

Using this YAML file, we can instruct Kas to launch the build:

kas build kas-flutter-example.yml

Storage flash process

Images and Nvidia tools to flash Jetson platforms are packaged into a tarball built and deployed as a target image into the folder build/tmp-glibc/deploy/images.

The SD card image for the Jetson Xavier NX can be flashed in two different ways:

  • from the target board,
  • from the host.

Here, we explain how to flash it from the target board.

Note: Each Jetson model has its own particular storage layout.

First, we need to extract the tegraflash archive:

mkdir tegraflash                                                                                                                                                                                                                          
cd tegraflash                                                                                                                                                                                                                             
tar -xvzf ../build/tmp-glibc/deploy/images/jetson-xavier-nx-devkit/core-image-minimal-jetson-xavier-nx-devkit.tegraflash.tar.gz

Moreover, to be able to flash the Jetson Xavier NX, it is required to switch it in recovery mode. For that it is necessary to connect a jumper between the 3rd and 4th pins from the right hand side of the “button header” underneath the back of the module (FRC and GND; see the labeling on the underside of the carrier board).

With this done, the module will power up in recovery mode automatically and will be visible from the host PC as an additional USB device:

lsusb |egrep 0955
Bus 003 Device 047: ID 0955:7e19 NVIDIA Corp.

Overall, here are the steps to follow to flash the SD card and the SPI Nand:

  • Start with your Jetson powered off.
  • Enable the recovery mode, as indicated above.
  • Connect the USB cable from your Jetson to your development host.
  • Insert an SD card into the slot on the module.
  • Power on the Jetson and put it into recovery mode.
  • Execute ./doflash.sh from the extracted tegraflash archive.

Finally, the target will boot.

Launch the Flutter application

Now, it is possible to start the Flutter Gallery application which is part of the core-image-minimal image, together with the Flutter stack, with the following command:

flutter-drm-eglstream-backend -b /usr/share/gallery

Customize a Yocto SDK

The OpenEmbedded build system can also be used to generate an application development SDK, that is a self-extracting tarball containing a cross-development toolchain, libraries and headers. This allows application developers to build, deploy and debug applications without having to do the OpenEmbedded build themselves.

It is possible to enrich the SDK’s sysroots with additional packages, through the variables TOOLCHAIN_HOST_TASK and TOOLCHAIN_TARGET_TASK.
That allows for example to extend the SDK with for example profiling tools, debug tools, symbols to be able to debug offline.

So, we used these variables to append the Flutter SDK and required dependencies to the Yocto SDK, to be able to cross-build Flutter applications with it.

TOOLCHAIN_HOST_TASK:append = " nativesdk-flutter-sdk"
TOOLCHAIN_TARGET_TASK:append = " gtk+3-dev"

To build the SDK with the same setup as the image previously built, we invoke Kas as follows:

kas shell kas-flutter-example.yml -c "bitbake -fc populate_sdk core-image-minimal"

Deploy the SDK

The SDKs built by OpenEmbedded are deployed in the folder build/tmp-glibc/deploy/sdk, so they can be extracted as follows to a folder:

build/tmp-glibc/deploy/sdk/oecore-x86_64-armv8a-toolchain-nodistro.0.sh -y -d ${destination}

To use the SDK, it is required to source the environment setup script that will set some cross-compile variables, like CC, LD, GDB, in the shell environment to develop or debug applications with SDK’s sysroots:

source <destination>/environment-setup-armv8a-oe-linux

Cross-build the Flutter gallery application

To illustrate how to use the SDK, let’s see how to build the Gallery Flutter application with the Yocto SDK. Before calling the flutter command, the SDK environment-setup script has been sourced and the following environment variables have been set:

  • FLUTTER_SDK: the path to the Flutter SDK into the Yocto SDK,
  • ENGINE_SDK: where the Flutter engine shall be built,
  • PATH: to extend the shell environment with Flutter tools provided by the SDK.

We can then retrieve the application source code:

git clone git@github.com:flutter/gallery.git
cd gallery
git checkout 9eb785cb997ff56c46e933c1c591f0a6f31454f6

Here, it is a workaround, that allows the flutter command line to correctly find the version of Flutter SDK:

export SDK_ROOT=/sysroots/x86_64-oesdk-linux/usr/share/flutter/sdk
git config --global --add safe.directory $SDK_ROOT
chmod a+rw $SDK_ROOT -R
rm -rf ${SDK_ROOT}/bin/cache/pkg/sky_engine/

Without the workaround above, the following error is raised:

The current Flutter SDK version is 0.0.0-unknown.
[...]
Failed to find the latest git commit date: VersionCheckError: Command exited with code 128: git -c log.showSignature=false log -n 1 --pretty=format:%ad --date=iso
Standard out:
Standard error: error: object directory build/downloads/git2/github.com.flutter.flutter.git/objects does not exist; check .git/objects/info/alternates
fatal: bad object HEAD
Returning 1970-01-01 01:00:00.000 instead.
[...]

Set the required environment variables and build the application for Linux:

export FLUTTER_SDK="${destination}/sysroots/x86_64-oesdk-linux/usr/share/flutter/sdk"
export PATH=${FLUTTER_SDK}/bin:$PATH
export ENGINE_SDK="./engine_sdk/sdk"

flutter config --enable-linux-desktop
flutter doctor -v
flutter build linux --release
flutter build bundle

This gives you the Flutter application, ready to run on the target!

Conclusion

In this blog post, we have shown that deploying Flutter on an OpenEmbedded distribution was a relatively easy process, and that the SDK can be extended to allow building Flutter applications.

Bootlin at the Embedded Linux Conference North America 2022

Bootlin CEO Thomas Petazzoni and COO Alexandre Belloni will both be attending the next Embedded Linux Conference North America, on June 21-24 in Austin, Texas.

In addition, both Thomas and Alexandre will be speaking at the event:

  • Thomas Petazzoni will give a talk Buildroot: what’s new?, providing an update on the improvements and new features in the Buildroot build system that have been integrated over the past two years
  • Alexandre Belloni will give a talk Yocto Project Autobuilders and the SWAT Team, during which he will explain what’s happening behind in the scenes in the Yocto Project to review and validate contributions before they are integrated.

Thomas and Alexandre will also naturally be available during the event to discuss business or career opportunities, so do not hesitate to get in touch if you’re interested.

Finally, prior to the event, Thomas Petazzoni will be in the Bay Area on June 13-15, also available for meetings or discussions.

Maintaining Yocto Project Documentation

For many years, Bootlin has been a strong user and a contributor to the Yocto Project, delivering numerous customized embedded Linux distributions and Board Support Packages based on Yocto Project and OpenEmbedded to its customers, for a wide range of hardware platforms and architectures.

In 2021, we have been able to bring this engagement further, as Bootlin engineer Michael Opdenacker has been given the opportunity to work as a maintainer for Yocto Project’s documentation, thanks to funding from the Linux Foundation. Since the mourning of Scott Rifenbark, the former maintainer, in early 2020, the project was in need for someone to fill this role.

Yocto Project developers and contributors did their best in the meantime though, in particular by migrating the documentation sources from the DocBook format to reStructuredText, to generate documentation using Sphinx. This migration was also done by the Linux kernel community, as reStructuredText seems to make it easier for developers to contribute to documentation.

For Michael Opdenacker, the interest of getting back to Yocto Project and OpenEmbedded was strong: he was one of the early users, over 15 years ago, of BitBake and OpenEmbedded. Many things have changed since Michael was generating his own OPIE and GPE Palmtop Environment images for handheld devices back in 2004 and 2005.

Indeed, since that time, under the Yocto Project umbrella, BitBake and OpenEmbedded have formed what’s probably the smartest, most versatile and most powerful embedded Linux build system. This build system takes a pretty steep learning curve though, and that’s why maintaining high quality documentation was a key focus from the start.

In addition to acting as a maintainer (reviewing and merging patches from contributors, encouraging contributions, managing documentation bugs, keeping the documentation current with the evolution of the code), we have so far also managed to:

  • Create brand new documentation, thanks to studying the source code and to knowledge from core developers. In particular, we documented vulnerability (CVE) management and drafted a first description of Hash Equivalence.
  • Implement a new infrastructure to generate diagrams from SVG sources. Currently, most graphics are bitmaps and do not render well in PDF and EPUB output. The next step is to migrate all diagrams to SVG.
  • Implement styling improvements, trying to eliminate unnecessarily complex sentences and make the documents easier to read.
  • Propose a “Documentation standards” document in markdown format, together with a diagram template with reusable shapes, text boxes and clipart, so that new documentation and diagrams are created in a way that’s consistent with existing documents.
  • Find bugs and improvement opportunities, and implement some of them, thanks to testing the software and reading the code.

Here are our contributions to the project so far, as of Oct. 20, 2021:

Yocto Project is a very welcoming open-source project, in which core developers are keen to help users and contributors. This makes it a real pleasure to participate to this project, in addition to the satisfaction to contribute to the progress of Embedded Linux.

If you are interested in studying Yocto Project and Bitbake’s documentation and contributing to it, you will be most welcome! The best is to get in touch with our community through the docs mailing list or chat with us on IRC. See our mailing lists and chat page for details.

If you’re interested in using Yocto Project for your embedded Linux projects, we also recommend you to check out our Yocto Project and OpenEmbedded system development training course, and its freely available training materials. We recently announced new dates for our upcoming online training classes on the Yocto Project:

Videos and slides from Bootlin talks at Embedded Linux Conference Europe 2020

The Embedded Linux Conference Europe took place online last week. While we definitely missed the experience of an in-person event, we strongly participated to this conference with no less than 7 talks on various topics showing Bootlin expertise in different fields: Linux kernel development in networking, multimedia and storage, but also build systems and tooling. We’re happy to be publishing now the slides and videos of our talks.

From the camera sensor to the user: the journey of a video frame, Maxime Chevallier

Download the slides: PDF, source.

OpenEmbedded and Yocto Project best practices, Alexandre Belloni

Download the slides: PDF, source.

Supporting hardware-accelerated video encoding with mainline Linux, Paul Kocialkowski

Download the slides: PDF, source.

Building embedded Debian/Ubuntu systems with ELBE, Köry Maincent

Download the slides: PDF, source.

Understand ECC support for NAND flash devices in Linux, Miquèl Raynal

Download the slides: PDF, source.

Using Visual Studio Code for Embedded Linux Development, Michael Opdenacker

Download the slides: PDF, source.

Precision Time Protocol (PTP) and packet timestamping in Linux, Antoine Ténart

Download the slides: PDF, source.

Covid-19: Bootlin proposes online sessions for all its courses

Tux working on embedded Linux on a couchLike most of us, due to the Covid-19 epidemic, you may be forced to work from home. To take advantage from this time confined at home, we are now proposing all our training courses as online seminars. You can then benefit from the contents and quality of Bootlin training sessions, without leaving the comfort and safety of your home. During our online seminars, our instructors will alternate between presentations and practical demonstrations, executing the instructions of our practical labs.

At any time, participants will be able to ask questions.

We can propose such remote training both through public online sessions, open to individual registration, as well as dedicated online sessions, for participants from the same company.

Public online sessions

We’re trying to propose time slots that should be manageable from Europe, Middle East, Africa and at least for the East Coast of North America. All such sessions will be taught in English. As usual with all our sessions, all our training materials (lectures and lab instructions) are freely available from the pages describing our courses.

Our Embedded Linux and Linux kernel courses are delivered over 7 half days of 4 hours each, while our Yocto Project, Buildroot and Linux Graphics courses are delivered over 4 half days. For our embedded Linux and Yocto Project courses, we propose an additional date in case some extra time is needed to complete the agenda.

Here are all the available sessions. If the situation lasts longer, we will create new sessions as needed:

Type Dates Time Duration Expected trainer Cost and registration
Embedded Linux (agenda) Sep. 28, 29, 30, Oct. 1, 2, 5, 6 2020. 17:00 – 21:00 (Paris), 8:00 – 12:00 (San Francisco) 28 h Michael Opdenacker 829 EUR + VAT* (register)
Embedded Linux (agenda) Nov. 2, 3, 4, 5, 6, 9, 10, 12, 2020. 14:00 – 18:00 (Paris), 8:00 – 12:00 (New York) 28 h Michael Opdenacker 829 EUR + VAT* (register)
Linux kernel (agenda) Nov. 16, 17, 18, 19, 23, 24, 25, 26 14:00 – 18:00 (Paris time) 28 h Alexandre Belloni 829 EUR + VAT* (register)
Yocto Project (agenda) Nov. 30, Dec. 1, 2, 3, 4, 2020 14:00 – 18:00 (Paris time) 16 h Maxime Chevallier 519 EUR + VAT* (register)
Buildroot (agenda) Dec. 7, 8, 9, 10, and 11, 2020 14:00 – 18:00 (Paris time) 16 h Thomas Petazzoni 519 EUR + VAT* (register)
Linux Graphics (agenda) Dec. 1, 2, 3, 4, 2020 14:00 – 18:00 (Paris time) 16 h Paul Kocialkowski 519 EUR + VAT* (register

* VAT: applies to businesses in France and to individuals from all countries. Businesses in the European Union won’t be charged VAT only if they provide valid company information and VAT number to Evenbrite at registration time. For businesses in other countries, we should be able to grant them a VAT refund, provided they send us a proof of company incorporation before the end of the session.

Each public session will be confirmed once there are at least 6 participants. If the minimum number of participants is not reached, Bootlin will propose new dates or a full refund (including Eventbrite fees) if no new date works for the participant.

We guarantee that the maximum number of participants will be 12.

Dedicated online sessions

If you have enough people to train, such dedicated sessions can be a worthy alternative to public ones:

  • Flexible dates and daily durations, corresponding to the availability of your teams.
  • Confidentiality: freedom to ask questions that are related to your company’s projects and plans.
  • If time left, possibility to have knowledge sharing time with the instructor, that could go beyond the scope of the training course.
  • Language: possibility to have a session in French instead of in English.

Online seminar details

Each session will be given through Jitsi Meet, a Free Software solution that we are trying to promote. As a backup solution, we will also be able to Google Hangouts Meet. Each participant should have her or his own connection and computer (with webcam and microphone) and if possible headsets, to avoid echo issues between audio input and output. This is probably the best solution to allow each participant to ask questions and write comments in the chat window. We also support people connecting from the same conference room with suitable equipment.

Each participant is asked to connect 15 minutes before the session starts, to make sure her or his setup works (instructions will be sent before the event).

How to register

For online public sessions, use the EventBrite links in the above list of sessions to register one or several individuals.

To register an entire group (for dedicated sessions), please contact training@bootlin.com and tell us the type of session you are interested in. We will then send you a registration form to collect all the details we need to send you a quote.

You can also ask all your questions by calling +33 484 258 097.

Questions and answers

Q : Should I order hardware in advance, our hardware included in the training cost?
R : No, practical labs are replaced by technical demonstrations, so you will be able to follow the course without any hardware. However, you can still order the hardware by checking the “Shopping list” pages of presentation materials for each session. This way, between each session, you will be able to replay by yourself the labs demonstrated by your trainer, ask all your questions, and get help between sessions through our dedicated Matrix channel to reach your goals.

Q: Why just demos instead of practicing with real hardware?
A: We are not ready to support a sufficient number of participants doing practical labs remotely with real hardware. This is more complicated and time consuming than in real life. Hence, what we we’re proposing is to replace practical labs with practical demonstrations shown by the instructor. The instructor will go through the normal practical labs with the standard hardware that we’re using.

Q: Would it be possible to run practical labs on the QEMU emulator?
R: Yes, it’s coming. In the embedded Linux course, we are already offering instructions to run most practical labs on QEMU between the sessions, before the practical demos performed by the trainer. We should also be able to propose such instructions for our Yocto Project and Buildroot training courses in the next months. Such work is likely to take more time for our Linux kernel course, practical labs being closer to the hardware that we use.

Q: Why proposing half days instead of full days?
A: From our experience, it’s very difficult to stay focused on a new technical topic for an entire day without having periodic moments when you are active (which happens in our public and on-site sessions, in which we interleave lectures and practical labs). Hence, we believe that daily slots of 4 hours (with a small break in the middle) is a good solution, also leaving extra time for following up your normal work.