I was invited to speak at the GENIVI All Members Meeting that took place on May 3-6 in Dublin, Ireland. This was a very interesting opportunity to meet new people in the In Vehicle Infotainment (IVI) industry and community.
In addition to the friendly social event at the Guiness Brewery, there was also a very interesting technical showcase of products and software using the GENIVI stack. I could observe that Freescale and ARM chips in general dominate this market. I also wore my Linaro shirt and had interesting discussions with several people about partnership opportunities between GENIVI and Linaro.
More and more feature rich Linux devices are put in the hands of consumers, and the average consumer shouldn’t even notice that they run Linux. To make the OS invisible, the system should boot in a flash.
Multiple boot time reduction techniques are now available, and can be used at the end of a development project, without incurring redesign costs. This presentation will guide embedded Linux system developers through the most effective ones. For each technique, we will detail how to use it and will report the exact savings achieved on a real embedded board.
Michael Opdenacker is the founder of Bootlin (https://bootlin.com), a company offering development, consulting and training services to embedded Linux system developers worldwide. He is always looking for innovative techniques to share with customers and with the community.
Michael is also the Community Manager for Linaro (http://linaro.org), a not-for-profit engineering organization working on software foundations for Linux on ARM, to reduce fragmentation between ARM chip vendors, increase product performance and reduce time to market. Linaro currently employs more than 100 of the most active developers in the ARM and embedded Linux community.
I was pleased to have a good number of participants, and to get many questions during and after the talk.
Though GENIVI is about Free and Open Source Software, it is unfortunately not very open to the community yet. You have to become a member to access its specifications, wiki and other technical resources. While collecting membership fees makes sense to operate such an organization, and is acceptable for system makers, it makes it difficult for embedded Linux community developers to get involved. I hope that GENIVI will become more open to the wider embedded Linux community in the future.
Just after the Embedded Linux Conference 2011, the first edition of the Android Builders Conference took place in San Francisco, on April 13th and April 14th 2011. This is the first, and to date, probably the first, conference entirely dedicated to Android low-level components and on how Android systems are built and modified. The number of resources, documentation and conferences on Android application development is already huge, but the amount of system-level information about Android is still relatively limited. This conference comes to fill in this gap, allowing engineers working on Android-based systems to share their experience. With a single track of talks for the first half-day, and two tracks for the second full day, it was a very nice first edition, and the co-location with the Embedded Linux Conference was well-appreciated. Interestingly enough, no talks were given by Google engineers, despite the fact that they are the primary designers and developers of the Android system.
Just as we did for the Embedded Linux Conference a few days ago, we are also publishing below the videos of all talks given during this Android Builders Summit. Of all the presentations, the ones we found the most interesting are certainly:
Karim Yaghmour’s talk about « Android Internals » and «Porting Android to new hardware»
Aleksander “Sasa” Gargenta’s talk «A walk through the Android stack». Unfortunately, the speaker had way too much contents for the one hour slot, but the content presented was very, very interesting.
Mark Brown’s talk «Linux audio for smartphones»
Mike Woster Linux Foundation Android Builders Summit Introduction Video (2 minutes): full HD (31M), 450×800 (11M)
Christy Wyatt Motorola Motorola: innovation rising Video (36 minutes): full HD (454M), 450×800 (142M)
Mark Charlebois Qualcomm Innovation Center From the alliance to the evolution: the history and future of Android innovation Video (26 minutes): full HD (332M), 450×800 (103M)
Greg Burns QuIC AllJoyn and the new era of peer-to-peer-technology Video (55 minutes): full HD (680M), 450×800 (209M)
Mark Brown Wolfson Micro Linux audio for smartphones Slides Video (46 minutes): full HD (560M), 450×800 (173M)
The Embedded Linux Conference 2011 took place between April, 11th and April, 13th in its now usual place, the Kabuki hotel in San Francisco, California. It was the first edition organized since the merge of the CE Linux Forum into the Linux Foundation. During three days, three parallel tracks of talks and BoFs about technical topics around embedded Linux : kernel support, power management, build systems, file systems, real-time, and more.
As usual, part of the Bootlin team was at this Embedded Linux Conference, in order to keep up with the latest developments from the embedded Linux community. Gregory Clement (left on the picture), Maxime Ripard (right on the picture) and myself (center on the picture) were present, and we recorded all talks of the conference. And just a little bit more than one month later, we are ready to announce that all videos are now available online, in 1080p high-definition, and in a lower 450p resolution, encoded with the new VP8 codec.
Amongst all the conferences below, each of us have selected the three ones we thought were the most interesting ones (note that the top three for each us is necessarily composed of distinct talks, as none of us have seen the same talks since we had to record talks from three different sessions in parallel) :
For Gregory Clement, the top three is: Yoshiya Hirase talk about Faster Resume For More Energy Savings on MeeGo, Arnd Bergmann talk about Optimizations For Cheap Flash Media (which follows Arnd article on the same topic in LWN) and a set of three related talks about the video infrastructure in the Linux kernel, that Gregory recommends to watch in this order: Media Controller Framework (MCF) For OMAP2+ Display Subsystem (Sumit Semwal), Video4linux: Progress, New videobuf2 Framework and the Future (Hans Verkuil) and Bringing up HDMI Display for OMAP4 Panda Board – Design, Challenges and Lessons Learned (Mythri pk).
For Maxime Ripard, the top three is: John Stultz talk about Android for servers, Mike Anderson talks about ARM NEON and GPU programming, Wolfram Sang talk about Helping the process
For myself, the top three is: Jesse Barker talks about the ARM Graphics ecosystem which gives a nice overview of the state of this topic, Hai Shalom talk about PCD (which is an original and interesting replacement for init), Dave Stewart talk about The Yocto Project and its Application Development Toolkit (because it gives details on how Yocto is supposed to be used for application development, a topic I’m interested in as a Buildroot developer)
It is also worth noting that this Embedded Linux Conference was co-located with the first edition of the Android Builders Summit, for which we will soon publish videos as well. The next embedded Linux conference will take place in Europe, in Prague from October 26th to 28th, co-located with the first edition of LinuxCon Europe and just after the Kernel Summit. Prague will really be full of Linux developers during this end of October, it’s time to book this week on your agenda as well !
Finally, the list of all videos of Embedded Linux Conference 2011, along with their corresponding slides :
Tim Bird Sony Network Entertainment Welcome Keynote Video (10 minutes): full HD (131M), 450×800 (43M)
Dirk Hohndel, Richard Purdie Intel, Linux Foundation The Yocto Project Video (35 minutes): full HD (458M), 450×800 (140M)
Keshava Munegowda Texas Instruments Power Fail Safe FAT File Systems Slides Video (48 minutes): full HD (693M), 450×800 (203M)
Frank Rowand Sony Identifying embedded real-time issues: I-cache and locks Slides Video (46 minutes): full HD (471M), 450×800 (147M)
Bruno Cardoso Lopes University of Campinas LLVM, Clang and Embedded Linux Systems Slides Video (50 minutes): full HD (593M), 450×800 (164M)
Steven Rostedt RedHat Kernel Shark Tutorial Video (49 minutes): full HD (743M), 450×800 (215M)
Kang Dongwook ETRI Snapshoot Booting on Embedded Linux Slides Video (33 minutes): full HD (284M), 450×800 (95M)
Khem Raj State of OpenEmbedded Internal Toolchain and SDKs Slides Video (41 minutes): full HD (289M), 450×800 (119M)
David Rusling Linaro Linaro: a year of change Slides Video (50 minutes): full HD (529M), 450×800 (173M)
Hai Shalom Atheros Control, recover and debug your embedded product with PCD Slides Video (50 minutes): full HD (470M), 450×800 (160M)
Gene Sally Zigbee Networking and Linux Video (53 minutes): full HD (262M), 450×800 (139M)
Xi Wang Broadcom Solving real-time scheduling problems with RT_PREEMPT and deadline-based scheduler Slides Video (43 minutes): full HD (422M), 450×800 (141M)
Mike Anderson The PTR Group ARM Neon instruction set and why you should care Slides Video (53 minutes): full HD (527M), 450×800 (169M)
Darren Hart Intel Yocto Project: Practical Kernel Development Tutorial Video (52 minutes): full HD (551M), 450×800 (196M)
Arnd Bergmann IBM Optimizations for cheap flash media Slides Video (49 minutes): full HD (482M), 450×800 (160M)
Wolfram Sang Pengutronix Developer’s diary: helping the process Slides Video (39 minutes): full HD (315M), 450×800 (112M)
Thomas Gleixner Linutronix RT-Preempt: what’s the state and why there is no roadmap Slides Video (46 minutes): full HD (447M), 450×800 (149M)
Jason Kridner Texas Instruments High-level web interface to low-level I/O on the BeagleBoard Slides Video (36 minutes): full HD (370M), 450×800 (115M)
Arnd Bergmann IBM Becoming part of the Linux kernel community Slides Video (34 minutes): full HD (376M), 450×800 (126M)
Paul Mundt Renesas Working with hardIRQs: life beyond static IRQ assignments Slides Video (36 minutes): full HD (330M), 450×800 (113M)
Amit Kucheria Linaro Powerdebugging inside Linaro Slides Video (46 minutes): full HD (309M), 450×800 (136M)
Mike Anderson The PTR Group High-performance computing using GPUs Slides Video (57 minutes): full HD (615M), 450×800 (185M)
Paul Larson Canonical Linaro automated validation on ARM Video (51 minutes): full HD (581M), 450×800 (184M)
Dave Stewart Intel The Yocto project and its application development toolkit (ADT) – The answer to effective embedded application development Video (42 minutes): full HD (362M), 450×800 (139M)
Damian Hobson Garcia, Katusya Matsubara, Takanari Hayama, Hisao Munakata Igel Integrating a Hardware Video Codec into Android Stagefright using OpenMAX IL Slides Video (55 minutes): full HD (564M), 450×800 (177M)
Koen Kooi Texas Instruments Integrating OpenEmbedded and Yocto Slides Video (52 minutes): full HD (465M), 450×800 (159M)
Mark Gross Intel How to power tune a device running on a Linux kernel for better suspend battery life Slides Video (49 minutes): full HD (273M), 450×800 (129M)
Remi Lorriaux Adeneo Embedded Real-time audio on embedded devices Slides Video (44 minutes): full HD (437M), 450×800 (138M)
Magnus Damm Runtime PM: upstream I/O device power management Slides Video (53 minutes): full HD (486M), 450×800 (164M)
Jesse Barker Linaro Linux graphics meets the ARM ecosystem Slides Video (50 minutes): full HD (329M), 450×800 (147M)
David Anders Texas Instruments Board bringup: open-source hardware and software tools Slides Video (38 minutes): full HD (376M), 450×800 (118M)
John Williams PetaLogix Dynamic co-simulation of FPGA-based systems on chip Slides Video (57 minutes): full HD (567M), 450×800 (198M)
Summit Semwal Texas Instruments Media Controller Framework (MCF) for OMAP2+ display subsystem Slides Video (49 minutes): full HD (518M), 450×800 (155M)
Anand Gadiyar Texas Instruments Tools and techniques for debugging embedded systems Slides Video (30 minutes): full HD (139M), 450×800 (81M)
Hans Verkuil Cisco Video4linux: progress, new videobuf2 framework and the future Slides Video (56 minutes): full HD (534M), 450×800 (171M)
Yoshiya Hirase Nokia Faster resume for more energy saving on MeeGo Slides Video (58 minutes): full HD (727M), 450×800 (218M)
Jake Edge Linux Weekly News What embedded Linux developers should know about IPv6 Slides Video (46 minutes): full HD (266M), 450×800 (122M)
Grégoire Gentil Always Innovating Hot multi-OS switch: how to run Ubuntu, ChromiumOS, Android at the same time on an embedded device Video (61 minutes): full HD (515M), 450×800 (174M)
Xi Wang Broadcom Controlling memory footpring at all layers: Linux kernel, applications, libraries and toolchain Slides Video (38 minutes): full HD (511M), 450×800 (152M)
Tom Zanussi, Saul Wold Building custom embedded images with Yocto Slides Video (49 minutes): full HD (500M), 450×800 (173M)
Philip Balister Open SDR A high performance interface between the OMAP3 and a FPGA Slides Video (51 minutes): full HD (347M), 450×800 (149M)
Jean Pihet NewOldBits.com The evolution of tracing and profiling for power management and accelerators Slides Video (40 minutes): full HD (428M), 450×800 (133M)
Elizabeth Flanagan Intel Delivering predictability: the Yocto project autobuilder, automated sanity testing, license collection and build statistics tracking Slides Video (48 minutes): full HD (241M), 450×800 (133M)
Mythri pk Texas Instruments Bringing up HDMI display for OMAP4 Panda board: design, challenges and lessons learned Slides Video (40 minutes): full HD (363M), 450×800 (122M)
As every year, FOSDEM, the largest community-driven open source conference in Europe, took place early February in Brussels. And again, Bootlin was around with its HD camcorder, to record the conferences of interest for embedded developers. They are now available for download!
Embedded-oriented filesystems are a scattered world. Flash-optimized filesystems are less so. JFFS2 has been widely used but has several performance issues (mount time, especially, though CONFIG_SUMMARY and sumtool fixes that since 2.6.15). LogFS doesn’t seem to be actively maintained. The most active and promising flash filesystem is UBIFS. It runs on top of UBI (“Unsorted Block Images”), an abstraction layer for MTD devices.
Why flash-oriented filesystems ?
MTDs (Memory Technology Devices) are very different from block devices: instead of a sequence of writable sectors, they contain an array of writable pages, organized in so-called “erase blocks”.
To write on a page that already has data on it, you first have to erase this data. However, it is only possible to erase whole eraseblocks. Only then, you can write your new data (including what you didn’t change). Erasing causes the memory cells to wear out. At some point, they won’t be usable anymore and have to be skipped.
Because it is memory-based, random access is theoretically as fast as sequential access. So, you don’t need to keep the fragments of your files together. It makes it possible to do wear-leveling and thus, “increase” the lifetime of the chip.
A simple way of doing wear-leveling is to keep track of the number of times a block has been erased and use the block that has been the least erased when updating data.
All these constraints make it hard to write a flash filesystem.
UBI intends to deal with all MTD-specific operations while still presenting random-access volumes to the the upper-layer. The first – and as for now, only – implementation using UBI is UBIFS. UBI is a “volume manager” and maps physical erase blocks (PEB) to logical erase blocks (LEB). The LEBs are smaller than the PEBs because of meta-data and headers.
How to use UBI on my board ?
There are mainly 2 ways to do that:
On a booted Linux system, approximately the same way you would create a partition on your desktop’s hard drive ;
From the bootloader, by flashing a previously prepared UBI image ;
Whatever solution you choose, you need to know the sizes of:
the eraseblocks (PEB) ;
the pages (or “minimum input/output size”) ;
the subpages (it may be the same as the min i/o size) ;
From these details, you can deduce another one: the size of logical erase blocks. It is the size of the PEB minus a data offset which is:
(subpage+page truncated to page size). This formula makes some assumption but should be correct if the subpage size is more than 8B and the page size more than 64B (see the source for more information). The best way to be sure of this size is to use mtdinfo on linux on the board. mtdinfo is part of the ubi-utils (part of mtd-utils). It’s probably available in your build system.
In both cases, you will also need a UBIFS image. In the way of JFFS2, mkfs.ubifs comes in mtd-utils (thus, you also need them on your desktop. Warning: mtd-utils in Ubuntu 10.10 are reported to be buggy ; if you use this distribution, recompile them from their git tree). Here is an example of how you can invoke it:
Let’s examine each command. ubiformat erases an MTD partition but keeps its erase counters ((‘X’ is the number of the partition you want to use). ubiattach creates a UBI device from the MTD partition. This UBI device is then referred to by UBI as ubi0 (if it is the first device). ubimkvol creates a volume on a UBI device ; this volume is referred to as ubi0_0 (if it is the first volume on the device). ubiupdatevol puts an image on an empty volume. (use ubiupdatevol -t /dev/ubi0_0 to empty a volume). At last, the well-known mount can be invoked using <device>:<volume>
Solution B – Prepare a UBI image ready to be flashed
It is more common to directly flash filesystem images directly from the bootloader. It is made possible by ubinize to prepare a UBI device image containing one or more volumes.
ubinize reads a configuration file (in the very simple INI format) describing the volumes and their configuration. Here is an example of a device with two volumes ; one, named rootfs is read-only (static), the other one, data is read-write (dynamic) ; the autoresize flag makes UBI resize to volume to use the whole unused space at initialization. The name of the sections is totally arbitrary.
For instance, with the previous examples and assuming the UBI device has been created/flashed on /dev/mtd1:
ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs
Creating and using a UBIFS filesystem is not as easy as with JFFS2 but UBI/UBIFS is designed to be more robust and UBI will ease the development of new filesystems. The authors of UBI have pointed some memory usage scalability problems but if a second version of UBI were written, filesystems on top of it would not need to be modified.
In case your system is missing the /dev/ubi_ctrl, /dev/ubi0 or /dev/ubi0_X device files, we advise you to recompile your kernel with DEVTMPFS and DEVTMPFS_MOUNT. This way, all the devices existing on your system will appear in /dev.
If you get write errors (code -74 or -5, probably), check that CONFIG_MTD_NAND_VERIFY_WRITE (respectively, ONENAND) is disabled : verifying subpages writes isn’t supported yet.
The primary place for information about MTD support in Linux is infradead.org. There also is a mailing list which you can also subscribe to.
The kernel sources under drivers/mtd and fs/ubisfs are also very helpful.
The Libre Software Meeting (Rencontres Mondiales du Logiciel Libre in French) is a community-driven event that takes place every year in France, and covers a wide range of topics in the free and open source software domain. Each year, an Embedded systems and free hardware topic is proposed, which in 2010 was lead by Florian Fainelli, Pierre Ficheux and myself.
While most of the talks took place in French, a few talks were given in English and as we recoded videos from those talks, we thought it’d be a good idea to highlight them to the english readers of our blog. We found it especially important since amongst those videos, there are two particularly interesting presentations from Sarah Sharp, a kernel developer from Intel, about USB3 and its support in Linux. As usual, all our videos are published under the Creative Commons Attribution – ShareAlike Licence version 3.0 license.
There are several ways to build a root filesystem for an embedded Linux system: Buildroot and Open Embedded are the usual solutions to do this. They allow to fine tune the contents of your filesystem. The drawback is, in both cases, that you need to build everything from sources and this can take from tens of minutes to several hours.
Sometimes you don’t need all this flexibility and you just want to have a ready-to-use root filesystem, to which you just add a few extra programs. In this case using a distribution is a good solution. So let’s see what we need:
A binary distribution
Available on several architectures
Ability to generate a “small” root filesystem
A large choice of packages
Oh, I think it is a pretty good description of Debian!
Emdebian is a project to adapt Debian to embedded devices. A good description from the Debian wiki is:
“In short, what EmDebian does is wrap around the regular Debian package building tools to provide a more fine grained control over package selection, size, dependencies and content to enable creation of very small and efficient Debian packages for use on naturally resource limited embedded targets.”
And so, pretty recently (2009), Emdebian released Multistrap which is similar to Debootstrap but more appropriate for embedded devices. It seems better by the way it builds a system:
It works in a completely different way by simply using apt and dpkg, rather than avoiding to use them, which is how Debootstrap works.
And also more appropriate by its goals:
It is focused on producing rootfs images for devices, as opposed to chroots for existing machines
Practical case: build a root filesystem for the USB A9263 board from Calao Systems (arm926ejs based board).
A drawback of Multistrap is its limitation to Debian, but in fact it is also usable on any distribution based on Debian. In our case, we ran it on an Ubuntu 10.04 system.
First, let’s install Multistrap:
$sudo apt-get install multistrap dpkg-dev
Multistrap needs a config file. For our needs we just use the example one given by Embedian. Let’s name it multistrap.conf:
# space separated package list
Grip is the name of the lightweight Debian distro built by Emdebian.
armel is the architecture, for our example: arm in little endian
-d $PWD/RFS is the output directory containing the root filesystem. Be careful to pass an absolute path.
-f multistrap.conf is the name of the configuration file
If you look at RFS/dev, you will see that there are no device files in it. This will be a problem at boot time, unless you build a kernel with the below options:
Thanks to these parameters, the kernel will automatically mount a tmpfs filesystem on /dev, and will populate it with devices present on the system.
Now we have a root filesystem, but we still need to run the package configuration scripts to make it usable. The packages were installed, but their configuration scripts couldn’t be executed, because they can only run on the target architecture.
The easiest way to do this is to use NFS. On the host side, you need to export the root filesystem directory through NFS. On the target side you have to select /bin/sh for the init process. A typical kernel command could be:
Install kernel modules (either manually or through a kernel package)
Add a new user
Create a minimum /etc/fstab file
You now have a Debian system for which it is very easy to add new software, and which can be configured in a very familiar way. That’s great to make product prototypes, small, low-power and secure servers for home or office use, and in some cases, even real products.
The program of the Embedded Linux Conference has been announced recently, and as usual, features a wide set of technical embedded Linux talks:
Filesystem/storage: Power Fail Safe FAT File System, Optimizations For Cheap Flash Media, from Arnd Bergmann, who has also recently published a very interesting article on the same topic.
Power management: Faster Resume For More Energy Savings on MeeGo, Powerdebug(ging): A Linaro Perspective, How to Power Tune a Device Running on a Linux Kernel for Better Suspend Battery Life, The Evolution of Tracing and Profiling for Power Management and Accelerators, Runtime PM: Upstream I/O Device Power Management
Real-time: Solving Real-Time Scheduling Problems with RT_PREEMPT and Deadline-Based Scheduler, Real-time Audio on Embedded Devices, Identifying Embedded Real-Time Latency Issues: I-Cache and Locks
Build system, with a huge number of Yocto-related talks, but no other build systems represented: State of OpenEmbedded Internal Toolchain and SDKs, Yocto Project: Practical Kernel Development Tutorial, Building Custom Embedded Images with Yocto, The Yocto Project and its Application Development Toolkit (ADT) – The Answer to Effective Embedded Application Development, Yocto Project Community BoFs, Delivering Predictability: The Yocto Project Autobuilder, Automated Sanity Testing, License Collection, and Build Statistics Tracking
FPGA: Dynamic Co-simulation of FPGA-based Linux Systems-on-Chip, A High Performance Interface Between the OMAP3 and an FPGA
Networking: What Embedded Linux Developers Should Know About IPv6, Zigbee Networking & Linux
Debugging: Kernel Shark Tutorial and Tools and Techniques for Debugging Embedded Systems
Optimization: Snapshot Booting on Embedded Linux, ARM Neon Instruction Set and Why You Should Care, Controlling Memory Footprint at All Layers: Linux Kernel, Applications, Libraries and Toolchain, High-Performance Computing using GPUs, What Are and How to Find a Program’s Unused DSOs
Low-level: Board Bringup: Open Source Hardware and Software Tools, Working with HardIRQs: Life Beyond Static IRQ Assignments, Genie in the Bottle: Linux Drivers for the AM1808 PRU
And many other talks on various topics: LLVM, Clang and Embedded Linux Systems, Linaro: A Year of Change, Control, Recover and Debug Your Embedded Product with PCD, Developer’s Diary: Helping the Process, High-Level Web Interface to Low-Level Linux I/O on the Beagleboard, Linaro Automated Validation on ARM, Crowd Sourcing and Protecting the Open Source Community, Android for Servers?, Hot Multi-OS Switch: How to run Ubuntu, ChromiumOS, Android at the Same Time on an Embedded Device.
This edition will be the first one organized since the merge between the CE Linux Forum into the Linux Foundation, and will therefore be a great opportunity to see if this merge had any impact on the technical quality of the conference.
My colleagues Maxime Ripard (who joined Bootlin just a week ago) and Gregory Clement as well as myself will be present at the Embedded Linux Conference and the Android Builders Summit, and we will as usual record all talks of both of these conferences and will put them online, as we have done recently for the talks that took place during the Embedded Linux Conference Europe 2010 in Cambridge. Do not hesitate to meet us in San Francisco!