Michael Opdenacker is the founder of Bootlin, and was its CEO until 2021. He is best known for all the free embedded Linux and kernel training materials that he created together with Thomas Petazzoni.
He is always looking for ways to increase performance,
reduce size and boot time, and to maximize Linux' world domination. More details...
If you are an embedded Linux developer too, you have probably been frustrated by the lack of information from the Linux kernel when it failed to start the init process when you’re building a new root filesystem. The only thing you get is No init found, and this could hide many different causes:
No init program candidate found at all
Some init program candidates exist but they can’t be executed, for multiple possible causes (missing execute permissions, failed to load shared libraries, executable compiled for an unknown architecture…)
The good news is that this source of frustration will be gone in Linux 3.13. Thanks to a Bootlin commit merged on Nov. 13, 2013, whenever an attempt to execute an init program candidate fails, there is a message in the console detailing the executable path and the error code. For example:
Starting init: /sbin/init exists but couldn't execute it (error -13)
When you get such a message, all you have to do is lookup the error code in include/uapi/asm-generic/errno-base.h or maybe in uapi/asm-generic/errno.h. In the above example, the -13 code meant permission denied, typically because of missing execution rights.
This had been annoying me for a long time, and I am glad that the Linux kernel community accepted my improvement!
With technical writing skills and an interest for training
We need to fill at least 2 open positions in the next months, and more will follow in 2014.
Newly graduated engineers are welcome too, provided they already have experience in the above technical fields or with Free Software development.
This time, we are looking for people who will be able to join one of our offices in France (Toulouse or Avignon), to strengthen our engineering teams there.
Toulouse is a dynamic city with lots of high-tech and embedded systems companies in particular. Our office in Colomiers can easily be reached by train from downtown Toulouse if you wish to settle there. You would be working with Maxime Ripard and our CTO Thomas Petazzoni.
Our main office is settled in Orange in the heart of the Provence region, close to Avignon, a smaller but dynamic city too. It enjoys a sunny climate and the proximity of the Alps and the Mediterranean sea. Accommodation is very affordable and there are no traffic issues! You would be working with our founder Michael Opdenacker and of course remotely with the rest of the engineering team. In particular, we are interested in foreign engineers who could help us develop our services in their home countries.
If you are unable to relocate this time, don’t hesitate to contact us anyway. Depending on your profile and experience, we are still planning to open home based jobs in a few months or years from now.
If you are interested in these positions, here are nice opportunities to meet us in the next weeks:
I found several people looking for a way to implement post-only mailing lists with GNU Mailman. However, I couldn’t find solutions that are described in sufficient detail.
In particular, this type of list is useful for notification mailing lists. In Bootlin’scase, whenever someone pushes commits to our public git trees, a notification e-mail is sent. Sometimes, internal discussions can follow, but we do not wish to make them public. This is why we do not want the list e-mail address to be shown in the messages that are sent. If the list address doesn’t appear in the To, CC or in Reply-To headers, members who are authorized to post messages without moderation won’t post replies to the list by mistake by using the “Reply to all” functionality of their e-mail client.
The problem is that the current version of GNU Mailman doesn’t support this type of list yet, at least with the parameters in the list administration interface. You can turn on the “Full personalization” option, which will send messages to each member individually, so that the list address doesn’t appear in the To header. You can also customize the Reply-To header, to an address that is different from the list address. However, the CC header will still hard-code the list address.
A possibility is to hack the /usr/lib/mailman/Mailman/Handlers/CookHeaders.py file, but this solution would apply to all the lists at once, and the changes you could make may interfere with Mailman updates. A much nice solution is to extend Mailman, to modify its behavior for specific mailing lists.
First, create a list-test mailing list. Some of the commands below will assume that you named your new list this way. Now, go to its administration interface and enable “Full Personalization” in “Non-digest” options. In “General options”, in the “Reply-To: header munging” section, specify a reply-to address.
If you send a test message to your new list, you will see that the list address is still in the CC header of the message that you receive.
Now, create a RemoveCC.py file in the Handlers directory (/usr/lib/mailman/Mailman/Handlers/RemoveCC.py on Ubuntu 12.04):
# Your comments here
"""Remove CC header in post-only mailing lists
This is to avoid unmoderated members to reply to messages,
making their replies public. Replies should instead go to a private list.
"""
def process(mlist, msg, msgdata):
del msg['Cc']
This will be yet another filter the list messages will go through. Now compile this file in the directory where you put it:
pycompile RemoveCC.py
The next thing to do is to modify the default filter pipeline for your new list. You can do it by creating a /var/lib/mailman/lists/list-test/extend.py file with the below contents:
import copy
from Mailman import mm_cfg
def extend(mlist):
mlist.pipeline = copy.copy(mm_cfg.GLOBAL_PIPELINE)
# The next line inserts MyHandler after CookHeaders.
mlist.pipeline.insert(mlist.pipeline.index('CookHeaders') + 1, 'RemoveCC')
This will add your new filter right after the CookHeaders one. To enable this, you have to run:
You can now send a new test message, and you will see that the CC header is now gone.
Notes
Of course, you can reuse the same extend.py file for multiple mailing lists. However, the solution doesn’t work if you don’t put the file inside /var/lib/mailman/lists/list-name (distributions other than Ubuntu 12.04 may have different paths).
I didn’t manage to undo this change. The Mailman wiki gives a solution based on creating a file containing del mlist.pipeline and running /usr/sbin/config_list -i this-file list-name, but it didn’t work for me. Please post a comment below if you find a way to implement this, and return to “factory” settings.
Don’t hesitate to share other ways of implementing this kind of functionality!
The Bootlin team wishes you a Happy New Year for 2013, with success in your professional and personal projects, and in contributing to other people’s lives. We are taking this opportunity to give some news about Bootlin.
In 2012, Bootlin continued to work on multiple development projects. The main difference with 2011 is that the projects were much longer. Here are the most important ones:
Linux kernel code development, adding and maintaining support for Marvell Armada 370 and Armada XP ARM SoCs in the mainline Linux kernel. Months of engineering work! Our commits appear on git.kernel.org.
Linux kernel code development and toolchain work on a new i.MX28 computer-on-module from Crystalfontz, adding support for this system to the mainline Linux kernel. See the project page on Kickstarter!
Build system integration, bootloader and kernel driver development, system update mechanism improvements, and general embedded Linux development work.
Kernel driver development and upstreaming for AT91 analog to digital converters.
Boot time optimization and power management audit on a MIPS based point of sales terminal
Boot time reduction project on a ARM based point-of-sales development kit.
Embedded Linux system integration, development and support.
Through contract work or through direct contributions, 2012 gave us multiple opportunities to contribute to open-source projects, in particular:
195 patches to the Linux kernel, plus the ones which have been accepted by maintainers but haven’t been included by Linus Torvalds yet. See git.kernel.org for details.
448 patches to the Buildroot build system. See git.buildroot.net for details.
We also created a new Android system development course and delivered multiple sessions of it. It is a four days training course to understand the Android system architecture, how to build and customize an Android system for a given hardware platform, and how to extend the Android platform to take new hardware devices into account.
As in the previous years, we also gave presentations at international conferences:
Thanks to their contributions to the mainline Linux kernel on the ARM platform, Gregory Clement and Thomas Petazzoni have also been invited to the ARM minisummit at the Linux kernel summit in San Jose in August. They were involved in decision making for the next evolutions of the Linux kernel on the ARM architecture.
We also organized and participated to two “Buildroot developer days” events, one in Brussels in February after Fosdem, and one in Barcelona in November after ELC Europe.
We also continued to participate to the development of the community of Linaro, an engineering organization working on improving Linux on the ARM platform. Note that this involvement is now over, allowing Michael Opdenacker to get back to more technical projects.
Now, let’s talk about our plans for 2013.
We plan to continue to hire more engineers to meet growing demand for our development and training services. In particular, a new engineer is joining us in March.
We are also organizing several public training sessions in France, which dates are now available:
Android system development, Toulouse, Apr. 2-5 (French)
We also plan to announce several new training sessions. Being very busy with projects in 2012, we haven’t had time to make progress in the plans we announced one year ago:
Git training. A two day training session to clearly understand how to use the Git distributed version control system, both for internal projects and for contribution to open-source projects.
Linux kernel debugging, tracing and performance analysis course. A one to two day session to trace kernel execution, investigate bugs and performance issues.
Boot time reduction training. A one to two day workshop to learn and master the methodology and techniques to make your embedded Linux systems boot faster.
As we are only in the very early stages of planning and preparing these courses, don’t hesitate to take the opportunity to contact us to let us know your expectations and influence their contents, in case you are interested in such courses.
We will also continue to participate to the key technical conferences. In particular, Bootlin engineers will be present at the Android Builders Summit and the Embedded Linux Conference in San Francisco, and at Embedded Linux Conference Europe in Edinburgh in October. This participation to conferences allows Bootlin engineers to remain up-to-date with the latest developments in the embedded Linux area and to create useful contacts in the community. Do not hesitate to go to such conferences, develop your technical knowledge and to take the opportunity to meet us there!
Last but not least, we will try harder to really write this newsletter every quarter. In 2012, we were so busy with projects that we didn’t manage to release newsletters for Q3 and Q4.
You can follow Bootlin news by reading our blog (31 articles in 2012) and by following our quick news on Twitter.
We have delivered two seminars about Android during the last quarter of 2012. The seminars were held in Belfort and Grenoble, France, and were organized by Captronic, a French public program to support innovation in electronic systems.
This one day seminar targets people who wish to understand the constraints and implications of using Android in embedded products, and know the steps to follow. The seminar is led by Maxime Ripard, Bootlin’sAndroid expert. Maxime is also the creator of Bootlin’sAndroid system development course.
Agenda
Morning
General introduction to Android
Opportunities to use Android in embedded systems which are neither phones nor tablets
Details on Android’s architecture and how to customize it:
Source code and compiling
Android changes to the Linux kernel
Bootloaders for Android
Supporting new hardware
Android filesystem layout
Android native layers and calling a C program to access specific hardware
Introduction to application development
Customizing the system
Using adb (Android Debug Bridge) for debugging and device remote access
Advice and resources
Afternoon
Completing the morning presentations (if necessary)
Demonstrating multiple aspects of system development with Android:
Getting sources and compiling
Android emulator demonstration
Starting Android on an electronic board with an ARM OMAP3530 processor, using a serial console.
Adding support for specific buttons. “Back” button example.
Using adb: installing, accessing system logs, accessing a command line interface on the device, exchanging files with the PC.
Customizing the system: change the product name, the default wallpaper, add new properties.
To access specific hardware (such as a USB device), development of a native library and accessing this functionality from the Android framework through a specific class and JNI library.
Describing an application that allows to control a USB device.
If you are interested in having one of us run such a seminar on your own part of the world, giving the audience the opportunity to ask all the questions they can have on the use of Android in embedded systems, don’t hesitate to contact us.
Here’s a simple trick that I recently rediscovered when I worked on a boot time reduction project for a customer. It’s not rocket science, but you may not be aware of it.
Our customer was using fbv to display its logo right after the system booted. This is a way to show that the system is available while you’re starting the system’s main application:
fbv -d 1 /root/logo.bmp > /dev/null 2>&1
With Grabserial and using simple instrumentation with messages issued on the serial console before and after running the command, we found that this command was taking 878 ms to execute. The customer’s system had an AT91SAM9263 ARM SOC from Atmel, running at 200 MHz.
Even if fbv is a simple program (22 KB on ARM, compiled with shared libraries), decoding the logo image is still expensive. Here’s a way to get this compute cost out of your boot sequence. All you have to do is display your logo on your framebuffer, and then capture the framebuffer contents in a file:
fbv -d 1 /root/logo.bmp
cp /dev/fb0 /root/logo.fb
The new file is now a little bigger, 230400 bytes instead of 76990. However, displaying your boot logo can now be done by a simple copy:
This command now runs in only 54 ms. That’s only 6% of the initial execution time! The advantage of this approach is that it works with any kind of framebuffer pixel format, as long as you have at least one program that knows how to write to your own framebuffer.
Note that the dd command was used to read and write the logo in one shot, rather than copying in multiple chunks. We found that the equivalent cp and cat commands were slightly slower. Of course, the benchmark results will vary from one system to another. Our customer had heavily optimized their NOR flash access time. If you run this on a very slow storage device, using a much faster CPU, the time to display the logo may be several impacted by the time taken to read a bigger file from slower storage.
To get even better performance, another trick is to compress the framebuffer contents with LZO (supported by BusyBox), which is very fast at decompressing, and requires very little memory to run:
lzop -9 /root/logo.fb
The new /root/logo.fb.lzop file is now only 2987 bytes big. Of course, the compression rate will depend on your logo image. In our case, the splashscreen contains mostly white space and a simple monochrome company logo. The new command to put in your startup scripts is now:
lzopcat /root/logo.fb.lzo > /dev/fb0
The execution time is now just 52.5 ms! With a faster CPU, the time reduction would have been even bigger.
The ultimate trick for having a real and possibly animated splashscreen would be to implement your own C program, directly writing to the framebuffer memory in mmap() mode. Here’s a nice tutorial showing how easy it can be.
Note: this article was first written for the German edition of Linux Magazine, and was later posted in the English edition too. We negotiated the right to publish it on our blog after the print editions. Here is the original version (the paper versions were modified by the editors to make them more concise).
In the family tree of computers, personal computers (PCs) are the parents, while the children and teenagers are mobile devices. PCs are no longer physically attractive, getting close to retirement. They produce a lot of heat, and make all sorts of unpleasant noise when you are next to them. Noise is caused by keyboard presses, by fans that are essential to avoid computer meltdown, and by rotating disks that sound like nothing but something that rotates.
The last chance for this generation to survive a few more years is to send them to a remote place where nobody can see their old bodies and hear their annoying noise any more. This place is called The Cloud. Perhaps because it gets these systems closer to the final destination: heaven.
If you have a device that you feel like putting on your knees (without getting burned) and caress its skin (oops screen), and doesn’t make any noise but the pleasant sounds that you feel like listening too, chances are you have a device from the last generation.
One reason why your device doesn’t make any unwanted sound is because it doesn’t have rotating disks, but flash storage instead. Most modern devices have flash storage, and most of these devices run Linux. This article gives technical details about how Linux supports flash storage devices. It should mostly interest people creating embedded and multimedia devices using the Linux kernel to get the best performance out of their hardware. People who wish to hack the devices they own should be interested too.
Flash storage
Flash storage, also called solid state, has multiple advantages over rotating storage. First, the absence of mechanical and moving parts eliminate noise, increase reliability and resistance to shock and vibrations, and also reduces heat dissipation as well as power consumption. Second, random access to data is also much faster, as you no longer have to move a disk head to the right location on the medium, which can take milliseconds.
Flash also has its shortcomings, of course. First, for the same price, you have about 10 times less solid state storage than rotating storage. This can be an issue with operating systems that require Gigabytes of disk space. Fortunately, Linux only needs a few MB of storage. Second, writing to flash storage has special constraints. You cannot write to the same location on a flash block multiple times without erasing the entire block, called an “erase block”. This constraint can also cause write speed to be much lower than read speed. Third, flash blocks can only withstand a rather limited number of erases (from a few thousand for today densest NAND flash to one million at best). This requires to implement hardware or software solutions, called “wear leveling”, to make sure that no flash block gets written to much too often that the others.
NOR flash was the first type of flash storage that was invented. NOR is very convenient as it allows the CPU to access each byte one by one, in random order. This way, the CPU can execute code directly from NOR flash. This is very convenient for bootloaders, which do not have to be copied to RAM before executing their code.
NAND flash is today’s most popular type of flash storage, as it offers more storage capacity for a much lower cost. The drawback is that NAND storage is on an external device, like rotating storage. You have to use a controller to access device data, and the CPU cannot execute code from NAND without copying the code to RAM first. Another constraint is that NAND flash devices can come out of the factory with faulty blocks, requiring hardware or software solutions to identify and discard bad blocks.
Two types of NAND flash storage are available today. The first type emulates a standard block interface, and contains a hardware “Flash Translation Layer” that takes care of erasing blocks, implementing wear leveling and managing bad blocks. This corresponds to USB flash drives, media cards, embedded MMC (eMMC) and Solid State Disks (SSD). The operating system has no control on the way flash sectors are managed, because it only sees an emulated block device. This is useful to reduce software complexity on the OS side. However, hardware makers usually keep their Flash Translation Layer algorithms secret. This leaves no way for system developers to verify and tune these algorithms, and I heard multiple voices in the Free Software community suspecting that these trade secrets were a way to hide poor implementations. For example, I was told that some flash media implemented wear leveling on 16 MB sectors, instead of using the whole storage space. This can make it very easy to break a flash device.
The second type is raw flash. The operating system has access to the flash controller, and can directly manage flash blocks. Counting the number of times a block has been erased is also possible (“block erase count”). The Linux kernel implements a Memory Technology Device (MTD) subsystem that allows to access and control the various types of flash devices with a common interface. This gives the freedom to implement hardware independent software to manage flash storage, in particular filesystems. Freedom and independence is something we have learned to care about in our community.
The Linux MTD architecture
Linux MTD partitions
The first thing you can do is access raw flash storage and partitions. It is similar to accessing raw block devices through devices files like /dev/sda (whole device) and /dev/sda1, /dev/sda2, etc. (partitions).
MTD devices are usually partitioned. This is useful to define areas for different purposes, such as:
Example MTD partitions
Raw means that no filesystem is used. This is not needed when you just have one binary to store, instead of multiple files.
Declaring partitions as read-only is also a way to make sure that Linux won’t allow to make changes to such partitions. This way, the bootloader and root filesystem partitions can be protected against mistakes and unauthorized modification attempts. You can also note that partitions cannot be bypassed by accessing the whole device at a given offset, as Linux offers no device file to access the whole storage.
What’s special in MTD partitions is that there is no partition table as in block devices. This is probably because flash is an unsafe location to store such critical system information, as flash blocks may become bad during system life.
Instead, partitions are defined in the kernel. An example is found in the arch/arm/mach-omap2/board-omap3beagle.c file in the kernel sources, defining flash partitions for the Beagle board:
Fortunately, you can override these default definitions without having to modify the kernel sources.
You first need to find the name of the MTD device to partition, as you may have multiple ones. Look at the
kernel log at boot time. In the Beagle board example, the MTD device name is omap2-nand.0:
Fortunately, you can define your own partitions without having to modify the kernel sources. The Linux kernel offers an mtdpartss boot parameter to define your own partition boundaries.
You can now add an mtdparts definition to the kernel command line (change it through the bootloader):
We have just defined 6 partitions in the omap2-nand.0 device:
First stage bootloader (128 KiB, read-only)
U-Boot (256 KiB, read-only)
U-Boot environment (128 KiB)
Kernel (4 MiB, read-only)
Root filesystem (16 MiB, read-only)
Data (remaining space)
Note that partition sizes must be a multiple of the erase block size. The erase block size can be found in /sys/class/mtd/mtdx/erasesize on the target system.
Now that partitions are defined, you can display the corresponding MTD devices by viewing /proc/mtd (the sizes are in hexadecimal):
Here, you can also see another difference with block devices. Device files names for block device partitions still refer to the complete device name (for example /dev/sda1 for the first partition of the device represented by /dev/sda). MTD partitions are shown as independent MTD devices, and for example mtd1 could either be the second partition of the first flash device, or the first partition of the second flash device. You cannot tell the difference from device names.
Back to our example, you can see that a separate flash partition is dedicated to storing the U-Boot environment variables. Did you know that you can update these variables from Linux, by flashing an image for this partition? At Bootlin, we have contributed a utility to create such an image.
Manipulating MTD devices
You can access MTD device number X through two types of interfaces. The first interface is a /dev/mtdX character device, managed by the mtdchar driver. In particular, this character device provides ioctl commands that are typically used by mtd-utils commands to manipulate and erase blocks in an MTD device.
The second interface is a /dev/mtdblockX block device, handled by the mtdblock driver. This device is mostly used to mount MTD filesystems, such as JFFS2 and YAFFS2, because the mount command primarily works with block devices. You may be tempted to use this device to write to the MTD device, but the corresponding driver isn’t elaborate enough for use in production. When you attempt to write to a given block, the previous contents are copied to RAM, the MTD block is erased, and the updated contents are written to the block. As you can see, there is no wear leveling of any sort, as a series of writes to the same part of the block device could very quickly damage the corresponding erase blocks. Worse, mtdblock isn’t even bad block aware. If you copy a filesystem image directly to /dev/mtdblockX, and your NAND storage has bad blocks, your filesystem will be corrupted because of the failure to write parts of the filesystem image.
Therefore, the clean way to manipulate MTD devices is through the character interface, and using the mtd-utils commands. Here are the most common ones:
mtdinfo to get detailed information about an MTD device
flash_eraseall to completely erase a given MTD device
These commands are available through the mtd-utils package in GNU/Linux distributions and can also be cross-compiled from source by embedded Linux build systems such as Buildroot and OpenEmbedded. Simple implementations of the most common commands are also available in BusyBox, making them much easier to cross-compile for simple embedded systems.
JFFS2
Journaling Flash File System version 2 (JFFS2), added to the Linux kernel in 2001, is a very popular filesystem for flash storage. As expected in a flash filesystem, it implements bad block detection and management, as well as wear leveling. It is also designed to stay in a consistent state after abrupt power failures and system crashes. Last but not least, it also stores data in compressed form. Multiple compressing schemes are available, according to whether matters more: read/write performance or the compression rate. For example, zlib compresses better than lzo, but is also much slower.
Implementing flash filesystems has special constraints. When you make a change to a particular file, you shouldn’t just go the easy way and copy the corresponding blocks to RAM, erase them, and flash the blocks with the new version. The first reason is that a power failure during the erase or write operations would cause irrecoverable data loss. The second reason is that you could quickly wear out specific blocks by making multiple updates to the same file.
Another solution is to copy the new data to a new block, and replace references to the old block by references to the new block. However, this implies another write on the filesystem, causing more references to be modified until the root reference is reached.
JFFS2 uses a log-structured approach to address this problem. Each file is described through a “node”, describing file metadata and data, and each node has an associated version number. Instead of making in-place changes, the idea is to write a more recent version of the node elsewhere in an erase block with free space. While this simplifies write operations, this complicates read ones, as reading a file requires to find the most recent node for this file.
To optimize performance, JFFS2 keeps an in-memory map of the most recent nodes for each file. However, this requires to scan all the nodes at mount time, to reconstitute this map. This is very expensive, as JFFS2’s mount time is proportional to the number of nodes. Embedded systems using JFFS2 on big flash partitions incurred big boot time penalties because of this. Fortunately, a CONFIG_JFFS2_SUMMARY kernel option was added, allowing to store this map on the flash device itself and dramatically reduce mount time. Be careful, this option is not turned on by default!
Back to node management, older nodes must be reclaimed at some point, to keep space free for newer writes. A node is created as “valid” and is considered as “obsolete” when a newer version is created. JFFS2 managed three types of flash blocks:
Clean blocks, containing only valid nodes
Dirty blocks, containing at least one obsolete node
Free blocks, not containing any node yet
JFFS2 runs a garbage collector in the background that recycles dirty blocks into free blocks. It does this by collecting all the valid nodes in a dirty block, and copying them to a clean block (with space left) or to a free block. The old dirty block is then erased and marked as free. To make all the erase blocks participate to wear leveling, the garbage collector occasionally consumes clean blocks too. See Wikipedia for more details about JFFS2.
There are two ways of using JFFS2 on a flash partition. The first way is to erase the partition and format it for JFFS2, and then mount it:
flash_eraseall -j /dev/mtd2
mount -t jffs2 /dev/mtdblock2 /mnt/flash
Note that flash_eraseall -j both erases the flash partition and formats it for JFFS2. You can then fill the partition by writing data into it.
The second way, which is more convenient to program production devices, is to prepare a JFFS2 image on a development workstation, and flash this image into the partition:
To prepare the JFFS2 image, you need to use the mkfs.jffs2 command supplied by mtd-utils. Do not be confused by its name: unlike some other mkfs commands, it doesn’t create a filesystem, but a filesystem image.
You first need to find the erase block size (as explained earlier). Let us assume it is 256 MiB.
-d specifies is a directory with the filesystem contents
--pad allows to create an image which size is a multiple of the erase block size.
--no-cleanmarkers should only be used for NAND flash.
It is fine to have a JFFS2 image that is smaller than the MTD partition. JFFS2 will still be able to use the whole partition, provided it was completely erased ahead of time.
Note that to prepare production devices, it is much more convenient to flash your MTD partitions from the bootloader, using a bad block aware command, without having to boot Linux. This way, you do not have to put development utilities such as flash_eraseall in the Linux root filesystem. This is another reason why filesystem images are useful. You typically download the filesystem image to RAM through the network, and then copy the image to flash. When you do this, just make sure that you copy the exact image size. With kernel images, we often copy a bigger number of bytes from RAM to flash, as the exact image size can vary, and this creates no issue. With JFFS2 images, if you copy more bytes from RAM to flash, you will end up writing flash with random bytes from RAM after the end of your image, which will corrupt the filesystem. I’m warning you because this is a typical mistake the people make during our training sessions.
YAFFS2
YAFFS2 is Yet Another Flash Filesystem which apparently was created as an alternative to JFFS2. It doesn’t use compression, but features a much faster mount time, as well as better read and write performance than JFFS2. YAFFS2 is available with a dual GPL and Proprietary license, GPL for use in the Linux kernel, and proprietary for proprietary operating systems. Revenue from the proprietary license allowed the fund the development of this filesystem.
YAFFS2 less popular than JFFS2, and this is probably because it is not part of the mainline Linux kernel. Instead, it is available as separate code with scripts to patch most versions of the Linux kernel source. There was an effort to get it mainlined about one year ago, but this attempt failed because the changes the kernel maintainers asked for would have broken the portability to other operating systems, and therefore would have compromised the project business model.
To use YAFFS2 after patching your kernel, you just need to erase your partition:
flash_eraseall /dev/mtd2
The filesystem is automatically formatted at the first mount:
mount -t yaffs2 /dev/mtdblock2 /mnt/flash
It is also possible to create YAFFS2 filesystem images with the mkyaffs tool, from yaffs-utils.
UBI and UBIFS
JFFS2 and YAFFS2 had a major issue: wear leveling was implemented by the filesystems themselves, implying that wear leveling was only local to individual partitions. In many systems, there are read-only partitions, or at least partitions that are very rarely updated, such as programs and libraries, as opposed to other read-write data areas which get most writes. These “hot” partitions take the risk of wearing out earlier than if all the flash sections participated in wear leveling. This is exactly what the Unsorted Block Images (UBI) project offers.
UBI is a layer on top of MTD which takes care of managing erase blocks, implementing wear leveling and bad block management on the whole device. This way, upper layers no longer have to take care of these tasks by themselves. UBI also supports flexible partitions or volumes, which can be created and resized dynamically, in a way that is similar to the Logical Volume Manager for block devices.
UBI works by implementing “Logical Erase Blocks” (LEBs), mapping to “Physical Erase Blocks” (PEBs). The upper layers only see LEBs. If an LEB gets written to too often, UBI can decide to swap pointers, to replace the “hot” PEB by a “cold” one. This mechanism requires a few free PEBs to work efficiently, and this overhead makes UBI less appropriate for small devices with just a few MB of space.
UBI Physical and Logical Erase Blocks
UBIFS is a filesystem for UBI. It was created by the Linux MTD project as JFFS2’s successor. It also supports compression and has much better mount, read and write performance.
The first way to use UBIFS is to initialize UBI from Linux:
Have /dev/ mounted as a devtmpfs filesystem
Erase your flash partition while preserving your erase counters
ubiformat /dev/mtd1
Attach UBI to one (of several) of the MTD partitions:
ubiattach /dev/ubi_ctrl -m 1
This command creates the ubi0 device, which represents the full UBI space stored on MTD device 1 (interfaced by a new /dev/ubi0 character device).
Create one or several volumes as in the below examples:
ubimkvol /dev/ubi0 -N test -s 116MiB
ubimkvol /dev/ubi0 -N test -m (max available size)
Mount an empty UBIFS filesystem on the new test volume:
mount -t ubifs ubi0:test /mnt/flash
You can then fill the filesystem by copying files to it
Note that it is also possible to create a UBIFS filesystem image with the mkfs.ubifs command and copy the image using ubiupdatevol.
The second way is to create an image of the entire UBI space, which can be flashed from the bootloader by a bad block aware command. To do this, first create a ubi.ini file describing the UBI space, its volumes and their contents. Here is an example:
You can then create the UBI image, for example specifying 128 KiB physical erase blocks and a minimum I/O size of 4096 bytes:
ubinize -o ubi.img -p 128KiB -m 4096 ubi.ini
The last steps are to flash the image file from the bootloader, using a bad block aware command, and add some parameters to the kernel command line:
ubi.mtd=1 (equivalent to ubiattach)
rootfstype=ubifs root=ubi0:rootfs if you use the UBIFS volume as root filesystem.
LogFS
As its name says, LogFS is another log-structured flash filesystem. It has an innovative design that could compete with UBIFS, and is now part of the mainline Linux kernel since version 2.6.34.
Unfortunately, the last time we tested it, LogFS was unstable and caused kernel oopses at unmount time. Therefore, we couldn’t compare it with the other filesystems. Being in the mainline Linux sources makes its code easier to maintain and fix though, and the bugs may be fixed in the latest kernel version when you read this article.
More details about LogFS can be found on Wikipedia.
SquashFS
For read-only partitions, it is actually possible to use the SquashFS block filesystem on MTD devices. My first idea was to directly copy a SquashFS image to the corresponding /dev/mtdblockx device. After all, this filesystem is read-only, and you don’t need any wear-leveling of any kind, as you never make any write. This worked very well, and I got very good performance results, until I tried to use SquashFS on a device that happened to have bad blocks. Remember that the mtdblock driver isn’t bad block aware. As a consequence, the SquashFS images didn’t get copied properly and the filesystem was corrupted. A bad block aware block device was therefore required.
There are two ways to do this. It is first possible to use the gluebi driver that emulates an MTD device on top of a UBI volume. As UBI discards bad blocks, it is then safe to use the mdtblock driver on top of this new MTD device.
A second possibility is to use the ubiblock driver (first submitted to the Linux Kernel Mailing List in 2011 by Bootlin, and revived by Ezequiel Garcia in November 2012, which implements a block device directly on top of UBI. Our benchmarks showed that this is a more efficient solution, as it doesn’t have to emulate an intermediate MTD device).
Benchmarks
Bootlin has run performance benchmarks to compare the various flash filesystems, with funding from the Linux foundation. The benchmarks and their results are described on eLinux.org.
These benchmarks showed that JFFS2 has the worst performance, and must absolutely be compiled with CONFIG_SUMMARY to have an acceptable boot time. However, JFFS2 is still the best compromise for devices with small flash partitions, for which compression is required, and where UBI would have too much space overhead. This is the reason why JFFS2 is still in use in OpenWRT, a distribution mainly targeting embedded devices like residential gateways and routers, with typically 4 to 16 MB of flash storage.
YAFFS2, thanks to improvements in the last years, shows very good if not best performance in many test scenarios. However, its drawbacks remain the lack of compression and its absence from the mainline Linux kernel sources. It also has weird performance issues managing directories.
UBIFS is now the best solution in terms of performance and space, except for small partitions in which its space overhead is significant. Its only drawback is that it requires a bit more work to deploy, compared to the other filesystems.
At the time of this writing, LogFS is too experimental to be used in production systems, though you can expect its bugs to be fixed over time, as its code is in the mainline kernel sources.
Last but not least, SquashFS can also be used on MTD flash, in systems with read-only partitions. This filesystem exhibits good compression, good mount time, and good read performance as well. The requirement to use SquashFS on top of UBI impairs its mount time performance though. On block filesystems, SquashFS exhibits the best mount time, but it looses a lot of time when it is on top of UBI, which takes a substantial amount of time to initialize (ubiattach operation).
The good news is that it is very cheap to switch filesystems. Applications won’t notice the difference. As our benchmarks have shown, you may get noticeable performance results, according to the size of your partitions, to the size and number of files, to the read and write patterns of your system, and to whether your files can be compressed or not. All you have to do is try the various filesystems, run your application and system tests, and keep the solution that maximizes performance for your particular system.
Back to flash storage with a block interface
We have seen the MTD subsystem and several filesystems allowing for complete control on the way flash blocks are managed. This allows to choose the wear leveling and block management scheme that best matches the various characteristics of the system.
But what to do when you are stuck with flash storage with a block interface, like SD cards for example? With these devices, you have no details about the erase block size and about the wear leveling algorithm. While these media are fine for external storage which just get occasional writes, you may run into deep trouble if you use these as primary storage in a system with intense I/O operations.
This issue is getting all the more critical as NAND flash is being replaced by eMMC in many recent embedded boards. eMMC is NAND flash with an MMC interface, but as opposed to MMC, is soldered on the board, to be immune from reliability issues caused by vibrations. The main advantage of eMMC is its unit price, making it more attractive than individual NAND chips produced in smaller quantities. Another advantage is that the block device is immediately available at boot time, without requiring any intervention and scanning from the operating system. Not having to manage bad blocks and wear leveling also keeps software simpler, of course at the cost of less control as we said. Some board makers, for example the engineers at CALAO Systems, even predict the extinction of raw flash in the next years. Raw flash may just be kept for specific industrial applications, but would then get very expensive because of low production volume.
Fortunately, we are not completely stuck with no clue about the internals of such flash devices. Arnd Bergmann has studied cheap flash media and has developed flashbench, a benchmarking tool to find their erase block size. This allows to optimize file system settings and get huge performance boosts on these flash media, and reduce the number of block erases. Arnd has described is work in a very interesting article on LWN.net.
Other than that, you are still stuck with an opaque wear leveling mechanism, and it’s always wise to use techniques to minimize the number of writes:
Do not put a swap area on flash storage
Whenever possible, mount your filesystems as read-only, or use read-only filesystems (SquashFS)
Keep volatile files such as log files and locks in RAM (tmpfs). You do not need to keep them across reboots anyway, and you do not want to create unnecessary disk activity because of them.
Conclusions and what to remember
If you develop or hack a device with raw flash, your best option is to use the JFFS2 filesystem for small partitions, with the CONFIG_SUMMARY option. For medium to very large partitions, UBIFS will be the best compromise in terms of speed, size and boot time. However, you may get slightly better performance with YAFFS2, but at the expense of size.
If you have a device with only flash storage with a block interface, for example an SD card, download flashbench from Arnd Bergmann and optimize the settings of your filesystems to get the best performance out of your storage, and optimize its lifetime.
If you reached this part of the article, you have the patience and interest required to contribute to the MTD subsystem of the Linux kernel. Contributions, code reviews and new ideas are welcome!
Our partner ISEE is famous for their IGEPv2 board that we use in our embedded Linux course. This board is both powerful (running at 1 GHz) and featureful (on-board WiFi and Bluetooth, many connectors and expansion capabilities).
The good news is that ISEE has started to develop a new IGEPv5 board, which will be based on the new OMAP5 processor from Texas Instruments. This processor features in particular 2 ARM Cortex A15 cores running at up to 2 GHz, DDR3 RAM support, USB3, full HD 3D recording, and supporting 4 displays and cameras at the same time. Can you imagine what systems you could create with such a CPU?
If you are interested in such a board, it is still time for you to give them your inputs and expectations.
What should a perfect OMAP5 board be like? Don’t hesitate to leave your comments on this blog post. Be sure that ISEE will pay attention to them.
We are just returning from Barcelona, Spain, after participating to the 2012 edition of the Embedded Linux Conference Europe. My colleague Thomas Petazzoni has delivered the below presentation:
Your New ARM SoC Linux Support Check-List
Since Linus Torvalds raised warnings about the state of the ARM architecture support in the Linux kernel, a huge amount of effort and reorganization has happened in the way Linux supports ARM SoCs. From the addition of the device tree to the pinctrl subsystem, from the new clock framework to the new rules in code organization and design, the changes have been significant over the last one and half year in theARM Linux kernel world.
Based on the speaker’s experience on getting the new Marvell Armada 370 and Armada XP SoC supported in the mainline Linux kernel, we will give an overview of those changes and summarize the new rules for ARM Linux support. We aim at helping developers willing to add suppot for new ARM SoCs in the Linux kernel by providing a check-list of things to do.
Thomas Petazzoni is an embedded Linux engineer and trainer at Bootlin since 2008. He has been involved with multiple projects around the Linux kernel, especially the mainlining of Marvell Armada 370/XP SoCs support. He is also a major contributor to the Buildroot embedded Linux build system with more than 1100 patches merged.
The presentation slides and their sources are now available here. We have also shot a video of Thomas’ talk and it should be available in the next weeks. Stay tuned!
Do not hesitate to contact us if you are looking for engineers to port Linux to new hardware.
The software patent threat is back in the European Union, through the “Unitary Patent” project currently reviewed by the Legal Affairs Committee (JURI) of the European Parliament. In a few words, the idea is to let the European Patent Office (EPO) define what can be patented and what cannot be. The problem is the EPO is well known for supporting software patents, and in addition, there is no democratic control over it.
After the latest battles against software patents in 2005, it was time for me to take my pen again and try to warn our representatives about the threats from software patents. The below letter was sent last week to each of the JURI committee, in English or in French version.
It doesn’t constitute any in-depth legal study on software patents, because I am only an engineer, without any advanced legal knowledge. It is rather a testimonial of my worries about these patents. These worries are justified by multiple deviances that software patents caused over the years in the whole world, and by the constant pressure faced by our representatives to make such patents legal in the European Union.
It may not be too late to write to your representatives at the European Parliament, but in any case, it isn’t too late to sign the petition that many companies and individuals already signed.
Dear Member of the European Parliament,
I am the creator and General Manager of Bootlin, a young European engineering company specializing in embedded software, supporting worldwide companies in designing embedded systems, on a rapidly growing market.
It’s the availability of a great number of Open-Source building blocks that allowed our company to experience continuous growth since its inception in 2004. Many industrial and consumer electronics products are designed with these building blocks. These blocks are developed by a vibrant community of software developers in the whole world, which our company participates to.
This dynamism would have been reduced if software patents had been legal in the E.U., as they are in the U.S.A. and in Japan. Because of their great number and because they are often trivial, such patents constitute a real “minefield” for inventors creating software and systems embedding software. For a company with limited resources, it is indeed impossible to make sure that the ideas that they implement by programming, or the software components that they reuse, do not step on a method already patented by someone else. The creator of an innovative product including software then runs the risk of having his/her investment ruined by a bigger competitor threatened by this invention. This competitor, if it owns a sufficiently big patent pool, could always find a trivial software patent that the competing product would infringe, and have the distribution of this product stopped. Another danger comes from “Patent troll” companies that do not create any product and only hunt for companies with products that could that infringe the patents they own.
We are also worried by the fact that at least in the software industry, patents are deviated from their primary purpose of fostering innovation. It is the exact opposite that happens, and it seems that patents today are only instruments for giant corporations to fight against competitors, big or small, and to prevent them from distributing competing products. In addition, the first patents granted a temporary monopoly in exchange for revealing a secret manufacturing process. For many software patents, like the famous “double click” patent owned by Microsoft, there is no more secret to reveal, as their effects are very easy to reproduce. However, we continue to grant a monopoly to their owners.
Therefore, our company is worried about the current plans to set up a unitary patent with a flanking unified patent court.
We are concerned that the regulation on the unitary patent, as agreed in December 2011 by the negotiators of the Council, the Commission, and the Committee on Legal Affairs of the European Parliament, leaves any and every issue on the limits of patentability to the European Patent Office (EPO)’s case law, without any democratic control or review by an independent court.
However, in spite of the rejection of software patentability by the European Parliament, through its vote on September 24th, 2003 and July 6th, 2005, the EPO continued to grant software patents, under the deceiving term of “computer-implemented inventions”. That’s perhaps because the EPO has a financial interest in granting as many patents as possible, and therefore fuel an increase in the number of litigation cases, for the profit of attorney offices but discouraging innovation, which is the main driver for our modern economy.
The regulation on the unitary patent is an opportunity for the EU legislators to harmonize substantive patent law in the EU institutional and jurisdictional framework, and to put an end to the EPO’s self-motivated practices extending the realm of patentability to software. Failing to do so, this unitary patent will do more harm than good to the EU ICT firms.
For these reasons, we urge MEPs to adopt amendments which clearly state that the EPO’s decisions are subject to a review from the Court of Justice of the European Union, and which reaffirm the rejection of software patentability, as voted by the European Parliament.
We have delivered two seminars about Android during the last quarter of 2012. The seminars were held in Belfort and Grenoble, France, and were organized by 
Presentation slides are available in 
Flash storage, also called solid state, has multiple advantages over rotating storage. First, the absence of mechanical and moving parts eliminate noise, increase reliability and resistance to shock and vibrations, and also reduces heat dissipation as well as power consumption. Second, random access to data is also much faster, as you no longer have to move a disk head to the right location on the medium, which can take milliseconds. 
