Here’s a simple trick that I recently rediscovered when I worked on a boot time reduction project for a customer. It’s not rocket science, but you may not be aware of it.
Our customer was using fbv to display its logo right after the system booted. This is a way to show that the system is available while you’re starting the system’s main application:
fbv -d 1 /root/logo.bmp > /dev/null 2>&1
With Grabserial and using simple instrumentation with messages issued on the serial console before and after running the command, we found that this command was taking 878 ms to execute. The customer’s system had an AT91SAM9263 ARM SOC from Atmel, running at 200 MHz.
Even if fbv is a simple program (22 KB on ARM, compiled with shared libraries), decoding the logo image is still expensive. Here’s a way to get this compute cost out of your boot sequence. All you have to do is display your logo on your framebuffer, and then capture the framebuffer contents in a file:
fbv -d 1 /root/logo.bmp
cp /dev/fb0 /root/logo.fb
The new file is now a little bigger, 230400 bytes instead of 76990. However, displaying your boot logo can now be done by a simple copy:
This command now runs in only 54 ms. That’s only 6% of the initial execution time! The advantage of this approach is that it works with any kind of framebuffer pixel format, as long as you have at least one program that knows how to write to your own framebuffer.
Note that the dd command was used to read and write the logo in one shot, rather than copying in multiple chunks. We found that the equivalent cp and cat commands were slightly slower. Of course, the benchmark results will vary from one system to another. Our customer had heavily optimized their NOR flash access time. If you run this on a very slow storage device, using a much faster CPU, the time to display the logo may be several impacted by the time taken to read a bigger file from slower storage.
To get even better performance, another trick is to compress the framebuffer contents with LZO (supported by BusyBox), which is very fast at decompressing, and requires very little memory to run:
lzop -9 /root/logo.fb
The new /root/logo.fb.lzop file is now only 2987 bytes big. Of course, the compression rate will depend on your logo image. In our case, the splashscreen contains mostly white space and a simple monochrome company logo. The new command to put in your startup scripts is now:
lzopcat /root/logo.fb.lzo > /dev/fb0
The execution time is now just 52.5 ms! With a faster CPU, the time reduction would have been even bigger.
The ultimate trick for having a real and possibly animated splashscreen would be to implement your own C program, directly writing to the framebuffer memory in mmap() mode. Here’s a nice tutorial showing how easy it can be.
Note: this article was first written for the German edition of Linux Magazine, and was later posted in the English edition too. We negotiated the right to publish it on our blog after the print editions. Here is the original version (the paper versions were modified by the editors to make them more concise).
In the family tree of computers, personal computers (PCs) are the parents, while the children and teenagers are mobile devices. PCs are no longer physically attractive, getting close to retirement. They produce a lot of heat, and make all sorts of unpleasant noise when you are next to them. Noise is caused by keyboard presses, by fans that are essential to avoid computer meltdown, and by rotating disks that sound like nothing but something that rotates.
The last chance for this generation to survive a few more years is to send them to a remote place where nobody can see their old bodies and hear their annoying noise any more. This place is called The Cloud. Perhaps because it gets these systems closer to the final destination: heaven.
If you have a device that you feel like putting on your knees (without getting burned) and caress its skin (oops screen), and doesn’t make any noise but the pleasant sounds that you feel like listening too, chances are you have a device from the last generation.
One reason why your device doesn’t make any unwanted sound is because it doesn’t have rotating disks, but flash storage instead. Most modern devices have flash storage, and most of these devices run Linux. This article gives technical details about how Linux supports flash storage devices. It should mostly interest people creating embedded and multimedia devices using the Linux kernel to get the best performance out of their hardware. People who wish to hack the devices they own should be interested too.
Flash storage
Flash storage, also called solid state, has multiple advantages over rotating storage. First, the absence of mechanical and moving parts eliminate noise, increase reliability and resistance to shock and vibrations, and also reduces heat dissipation as well as power consumption. Second, random access to data is also much faster, as you no longer have to move a disk head to the right location on the medium, which can take milliseconds.
Flash also has its shortcomings, of course. First, for the same price, you have about 10 times less solid state storage than rotating storage. This can be an issue with operating systems that require Gigabytes of disk space. Fortunately, Linux only needs a few MB of storage. Second, writing to flash storage has special constraints. You cannot write to the same location on a flash block multiple times without erasing the entire block, called an “erase block”. This constraint can also cause write speed to be much lower than read speed. Third, flash blocks can only withstand a rather limited number of erases (from a few thousand for today densest NAND flash to one million at best). This requires to implement hardware or software solutions, called “wear leveling”, to make sure that no flash block gets written to much too often that the others.
NOR flash was the first type of flash storage that was invented. NOR is very convenient as it allows the CPU to access each byte one by one, in random order. This way, the CPU can execute code directly from NOR flash. This is very convenient for bootloaders, which do not have to be copied to RAM before executing their code.
NAND flash is today’s most popular type of flash storage, as it offers more storage capacity for a much lower cost. The drawback is that NAND storage is on an external device, like rotating storage. You have to use a controller to access device data, and the CPU cannot execute code from NAND without copying the code to RAM first. Another constraint is that NAND flash devices can come out of the factory with faulty blocks, requiring hardware or software solutions to identify and discard bad blocks.
Two types of NAND flash storage are available today. The first type emulates a standard block interface, and contains a hardware “Flash Translation Layer” that takes care of erasing blocks, implementing wear leveling and managing bad blocks. This corresponds to USB flash drives, media cards, embedded MMC (eMMC) and Solid State Disks (SSD). The operating system has no control on the way flash sectors are managed, because it only sees an emulated block device. This is useful to reduce software complexity on the OS side. However, hardware makers usually keep their Flash Translation Layer algorithms secret. This leaves no way for system developers to verify and tune these algorithms, and I heard multiple voices in the Free Software community suspecting that these trade secrets were a way to hide poor implementations. For example, I was told that some flash media implemented wear leveling on 16 MB sectors, instead of using the whole storage space. This can make it very easy to break a flash device.
The second type is raw flash. The operating system has access to the flash controller, and can directly manage flash blocks. Counting the number of times a block has been erased is also possible (“block erase count”). The Linux kernel implements a Memory Technology Device (MTD) subsystem that allows to access and control the various types of flash devices with a common interface. This gives the freedom to implement hardware independent software to manage flash storage, in particular filesystems. Freedom and independence is something we have learned to care about in our community.
The Linux MTD architecture
Linux MTD partitions
The first thing you can do is access raw flash storage and partitions. It is similar to accessing raw block devices through devices files like /dev/sda (whole device) and /dev/sda1, /dev/sda2, etc. (partitions).
MTD devices are usually partitioned. This is useful to define areas for different purposes, such as:
Example MTD partitions
Raw means that no filesystem is used. This is not needed when you just have one binary to store, instead of multiple files.
Declaring partitions as read-only is also a way to make sure that Linux won’t allow to make changes to such partitions. This way, the bootloader and root filesystem partitions can be protected against mistakes and unauthorized modification attempts. You can also note that partitions cannot be bypassed by accessing the whole device at a given offset, as Linux offers no device file to access the whole storage.
What’s special in MTD partitions is that there is no partition table as in block devices. This is probably because flash is an unsafe location to store such critical system information, as flash blocks may become bad during system life.
Instead, partitions are defined in the kernel. An example is found in the arch/arm/mach-omap2/board-omap3beagle.c file in the kernel sources, defining flash partitions for the Beagle board:
Fortunately, you can override these default definitions without having to modify the kernel sources.
You first need to find the name of the MTD device to partition, as you may have multiple ones. Look at the
kernel log at boot time. In the Beagle board example, the MTD device name is omap2-nand.0:
Fortunately, you can define your own partitions without having to modify the kernel sources. The Linux kernel offers an mtdpartss boot parameter to define your own partition boundaries.
You can now add an mtdparts definition to the kernel command line (change it through the bootloader):
We have just defined 6 partitions in the omap2-nand.0 device:
First stage bootloader (128 KiB, read-only)
U-Boot (256 KiB, read-only)
U-Boot environment (128 KiB)
Kernel (4 MiB, read-only)
Root filesystem (16 MiB, read-only)
Data (remaining space)
Note that partition sizes must be a multiple of the erase block size. The erase block size can be found in /sys/class/mtd/mtdx/erasesize on the target system.
Now that partitions are defined, you can display the corresponding MTD devices by viewing /proc/mtd (the sizes are in hexadecimal):
Here, you can also see another difference with block devices. Device files names for block device partitions still refer to the complete device name (for example /dev/sda1 for the first partition of the device represented by /dev/sda). MTD partitions are shown as independent MTD devices, and for example mtd1 could either be the second partition of the first flash device, or the first partition of the second flash device. You cannot tell the difference from device names.
Back to our example, you can see that a separate flash partition is dedicated to storing the U-Boot environment variables. Did you know that you can update these variables from Linux, by flashing an image for this partition? At Bootlin, we have contributed a utility to create such an image.
Manipulating MTD devices
You can access MTD device number X through two types of interfaces. The first interface is a /dev/mtdX character device, managed by the mtdchar driver. In particular, this character device provides ioctl commands that are typically used by mtd-utils commands to manipulate and erase blocks in an MTD device.
The second interface is a /dev/mtdblockX block device, handled by the mtdblock driver. This device is mostly used to mount MTD filesystems, such as JFFS2 and YAFFS2, because the mount command primarily works with block devices. You may be tempted to use this device to write to the MTD device, but the corresponding driver isn’t elaborate enough for use in production. When you attempt to write to a given block, the previous contents are copied to RAM, the MTD block is erased, and the updated contents are written to the block. As you can see, there is no wear leveling of any sort, as a series of writes to the same part of the block device could very quickly damage the corresponding erase blocks. Worse, mtdblock isn’t even bad block aware. If you copy a filesystem image directly to /dev/mtdblockX, and your NAND storage has bad blocks, your filesystem will be corrupted because of the failure to write parts of the filesystem image.
Therefore, the clean way to manipulate MTD devices is through the character interface, and using the mtd-utils commands. Here are the most common ones:
mtdinfo to get detailed information about an MTD device
flash_eraseall to completely erase a given MTD device
These commands are available through the mtd-utils package in GNU/Linux distributions and can also be cross-compiled from source by embedded Linux build systems such as Buildroot and OpenEmbedded. Simple implementations of the most common commands are also available in BusyBox, making them much easier to cross-compile for simple embedded systems.
JFFS2
Journaling Flash File System version 2 (JFFS2), added to the Linux kernel in 2001, is a very popular filesystem for flash storage. As expected in a flash filesystem, it implements bad block detection and management, as well as wear leveling. It is also designed to stay in a consistent state after abrupt power failures and system crashes. Last but not least, it also stores data in compressed form. Multiple compressing schemes are available, according to whether matters more: read/write performance or the compression rate. For example, zlib compresses better than lzo, but is also much slower.
Implementing flash filesystems has special constraints. When you make a change to a particular file, you shouldn’t just go the easy way and copy the corresponding blocks to RAM, erase them, and flash the blocks with the new version. The first reason is that a power failure during the erase or write operations would cause irrecoverable data loss. The second reason is that you could quickly wear out specific blocks by making multiple updates to the same file.
Another solution is to copy the new data to a new block, and replace references to the old block by references to the new block. However, this implies another write on the filesystem, causing more references to be modified until the root reference is reached.
JFFS2 uses a log-structured approach to address this problem. Each file is described through a “node”, describing file metadata and data, and each node has an associated version number. Instead of making in-place changes, the idea is to write a more recent version of the node elsewhere in an erase block with free space. While this simplifies write operations, this complicates read ones, as reading a file requires to find the most recent node for this file.
To optimize performance, JFFS2 keeps an in-memory map of the most recent nodes for each file. However, this requires to scan all the nodes at mount time, to reconstitute this map. This is very expensive, as JFFS2’s mount time is proportional to the number of nodes. Embedded systems using JFFS2 on big flash partitions incurred big boot time penalties because of this. Fortunately, a CONFIG_JFFS2_SUMMARY kernel option was added, allowing to store this map on the flash device itself and dramatically reduce mount time. Be careful, this option is not turned on by default!
Back to node management, older nodes must be reclaimed at some point, to keep space free for newer writes. A node is created as “valid” and is considered as “obsolete” when a newer version is created. JFFS2 managed three types of flash blocks:
Clean blocks, containing only valid nodes
Dirty blocks, containing at least one obsolete node
Free blocks, not containing any node yet
JFFS2 runs a garbage collector in the background that recycles dirty blocks into free blocks. It does this by collecting all the valid nodes in a dirty block, and copying them to a clean block (with space left) or to a free block. The old dirty block is then erased and marked as free. To make all the erase blocks participate to wear leveling, the garbage collector occasionally consumes clean blocks too. See Wikipedia for more details about JFFS2.
There are two ways of using JFFS2 on a flash partition. The first way is to erase the partition and format it for JFFS2, and then mount it:
flash_eraseall -j /dev/mtd2
mount -t jffs2 /dev/mtdblock2 /mnt/flash
Note that flash_eraseall -j both erases the flash partition and formats it for JFFS2. You can then fill the partition by writing data into it.
The second way, which is more convenient to program production devices, is to prepare a JFFS2 image on a development workstation, and flash this image into the partition:
To prepare the JFFS2 image, you need to use the mkfs.jffs2 command supplied by mtd-utils. Do not be confused by its name: unlike some other mkfs commands, it doesn’t create a filesystem, but a filesystem image.
You first need to find the erase block size (as explained earlier). Let us assume it is 256 MiB.
-d specifies is a directory with the filesystem contents
--pad allows to create an image which size is a multiple of the erase block size.
--no-cleanmarkers should only be used for NAND flash.
It is fine to have a JFFS2 image that is smaller than the MTD partition. JFFS2 will still be able to use the whole partition, provided it was completely erased ahead of time.
Note that to prepare production devices, it is much more convenient to flash your MTD partitions from the bootloader, using a bad block aware command, without having to boot Linux. This way, you do not have to put development utilities such as flash_eraseall in the Linux root filesystem. This is another reason why filesystem images are useful. You typically download the filesystem image to RAM through the network, and then copy the image to flash. When you do this, just make sure that you copy the exact image size. With kernel images, we often copy a bigger number of bytes from RAM to flash, as the exact image size can vary, and this creates no issue. With JFFS2 images, if you copy more bytes from RAM to flash, you will end up writing flash with random bytes from RAM after the end of your image, which will corrupt the filesystem. I’m warning you because this is a typical mistake the people make during our training sessions.
YAFFS2
YAFFS2 is Yet Another Flash Filesystem which apparently was created as an alternative to JFFS2. It doesn’t use compression, but features a much faster mount time, as well as better read and write performance than JFFS2. YAFFS2 is available with a dual GPL and Proprietary license, GPL for use in the Linux kernel, and proprietary for proprietary operating systems. Revenue from the proprietary license allowed the fund the development of this filesystem.
YAFFS2 less popular than JFFS2, and this is probably because it is not part of the mainline Linux kernel. Instead, it is available as separate code with scripts to patch most versions of the Linux kernel source. There was an effort to get it mainlined about one year ago, but this attempt failed because the changes the kernel maintainers asked for would have broken the portability to other operating systems, and therefore would have compromised the project business model.
To use YAFFS2 after patching your kernel, you just need to erase your partition:
flash_eraseall /dev/mtd2
The filesystem is automatically formatted at the first mount:
mount -t yaffs2 /dev/mtdblock2 /mnt/flash
It is also possible to create YAFFS2 filesystem images with the mkyaffs tool, from yaffs-utils.
UBI and UBIFS
JFFS2 and YAFFS2 had a major issue: wear leveling was implemented by the filesystems themselves, implying that wear leveling was only local to individual partitions. In many systems, there are read-only partitions, or at least partitions that are very rarely updated, such as programs and libraries, as opposed to other read-write data areas which get most writes. These “hot” partitions take the risk of wearing out earlier than if all the flash sections participated in wear leveling. This is exactly what the Unsorted Block Images (UBI) project offers.
UBI is a layer on top of MTD which takes care of managing erase blocks, implementing wear leveling and bad block management on the whole device. This way, upper layers no longer have to take care of these tasks by themselves. UBI also supports flexible partitions or volumes, which can be created and resized dynamically, in a way that is similar to the Logical Volume Manager for block devices.
UBI works by implementing “Logical Erase Blocks” (LEBs), mapping to “Physical Erase Blocks” (PEBs). The upper layers only see LEBs. If an LEB gets written to too often, UBI can decide to swap pointers, to replace the “hot” PEB by a “cold” one. This mechanism requires a few free PEBs to work efficiently, and this overhead makes UBI less appropriate for small devices with just a few MB of space.
UBI Physical and Logical Erase Blocks
UBIFS is a filesystem for UBI. It was created by the Linux MTD project as JFFS2’s successor. It also supports compression and has much better mount, read and write performance.
The first way to use UBIFS is to initialize UBI from Linux:
Have /dev/ mounted as a devtmpfs filesystem
Erase your flash partition while preserving your erase counters
ubiformat /dev/mtd1
Attach UBI to one (of several) of the MTD partitions:
ubiattach /dev/ubi_ctrl -m 1
This command creates the ubi0 device, which represents the full UBI space stored on MTD device 1 (interfaced by a new /dev/ubi0 character device).
Create one or several volumes as in the below examples:
ubimkvol /dev/ubi0 -N test -s 116MiB
ubimkvol /dev/ubi0 -N test -m (max available size)
Mount an empty UBIFS filesystem on the new test volume:
mount -t ubifs ubi0:test /mnt/flash
You can then fill the filesystem by copying files to it
Note that it is also possible to create a UBIFS filesystem image with the mkfs.ubifs command and copy the image using ubiupdatevol.
The second way is to create an image of the entire UBI space, which can be flashed from the bootloader by a bad block aware command. To do this, first create a ubi.ini file describing the UBI space, its volumes and their contents. Here is an example:
You can then create the UBI image, for example specifying 128 KiB physical erase blocks and a minimum I/O size of 4096 bytes:
ubinize -o ubi.img -p 128KiB -m 4096 ubi.ini
The last steps are to flash the image file from the bootloader, using a bad block aware command, and add some parameters to the kernel command line:
ubi.mtd=1 (equivalent to ubiattach)
rootfstype=ubifs root=ubi0:rootfs if you use the UBIFS volume as root filesystem.
LogFS
As its name says, LogFS is another log-structured flash filesystem. It has an innovative design that could compete with UBIFS, and is now part of the mainline Linux kernel since version 2.6.34.
Unfortunately, the last time we tested it, LogFS was unstable and caused kernel oopses at unmount time. Therefore, we couldn’t compare it with the other filesystems. Being in the mainline Linux sources makes its code easier to maintain and fix though, and the bugs may be fixed in the latest kernel version when you read this article.
More details about LogFS can be found on Wikipedia.
SquashFS
For read-only partitions, it is actually possible to use the SquashFS block filesystem on MTD devices. My first idea was to directly copy a SquashFS image to the corresponding /dev/mtdblockx device. After all, this filesystem is read-only, and you don’t need any wear-leveling of any kind, as you never make any write. This worked very well, and I got very good performance results, until I tried to use SquashFS on a device that happened to have bad blocks. Remember that the mtdblock driver isn’t bad block aware. As a consequence, the SquashFS images didn’t get copied properly and the filesystem was corrupted. A bad block aware block device was therefore required.
There are two ways to do this. It is first possible to use the gluebi driver that emulates an MTD device on top of a UBI volume. As UBI discards bad blocks, it is then safe to use the mdtblock driver on top of this new MTD device.
A second possibility is to use the ubiblock driver (first submitted to the Linux Kernel Mailing List in 2011 by Bootlin, and revived by Ezequiel Garcia in November 2012, which implements a block device directly on top of UBI. Our benchmarks showed that this is a more efficient solution, as it doesn’t have to emulate an intermediate MTD device).
Benchmarks
Bootlin has run performance benchmarks to compare the various flash filesystems, with funding from the Linux foundation. The benchmarks and their results are described on eLinux.org.
These benchmarks showed that JFFS2 has the worst performance, and must absolutely be compiled with CONFIG_SUMMARY to have an acceptable boot time. However, JFFS2 is still the best compromise for devices with small flash partitions, for which compression is required, and where UBI would have too much space overhead. This is the reason why JFFS2 is still in use in OpenWRT, a distribution mainly targeting embedded devices like residential gateways and routers, with typically 4 to 16 MB of flash storage.
YAFFS2, thanks to improvements in the last years, shows very good if not best performance in many test scenarios. However, its drawbacks remain the lack of compression and its absence from the mainline Linux kernel sources. It also has weird performance issues managing directories.
UBIFS is now the best solution in terms of performance and space, except for small partitions in which its space overhead is significant. Its only drawback is that it requires a bit more work to deploy, compared to the other filesystems.
At the time of this writing, LogFS is too experimental to be used in production systems, though you can expect its bugs to be fixed over time, as its code is in the mainline kernel sources.
Last but not least, SquashFS can also be used on MTD flash, in systems with read-only partitions. This filesystem exhibits good compression, good mount time, and good read performance as well. The requirement to use SquashFS on top of UBI impairs its mount time performance though. On block filesystems, SquashFS exhibits the best mount time, but it looses a lot of time when it is on top of UBI, which takes a substantial amount of time to initialize (ubiattach operation).
The good news is that it is very cheap to switch filesystems. Applications won’t notice the difference. As our benchmarks have shown, you may get noticeable performance results, according to the size of your partitions, to the size and number of files, to the read and write patterns of your system, and to whether your files can be compressed or not. All you have to do is try the various filesystems, run your application and system tests, and keep the solution that maximizes performance for your particular system.
Back to flash storage with a block interface
We have seen the MTD subsystem and several filesystems allowing for complete control on the way flash blocks are managed. This allows to choose the wear leveling and block management scheme that best matches the various characteristics of the system.
But what to do when you are stuck with flash storage with a block interface, like SD cards for example? With these devices, you have no details about the erase block size and about the wear leveling algorithm. While these media are fine for external storage which just get occasional writes, you may run into deep trouble if you use these as primary storage in a system with intense I/O operations.
This issue is getting all the more critical as NAND flash is being replaced by eMMC in many recent embedded boards. eMMC is NAND flash with an MMC interface, but as opposed to MMC, is soldered on the board, to be immune from reliability issues caused by vibrations. The main advantage of eMMC is its unit price, making it more attractive than individual NAND chips produced in smaller quantities. Another advantage is that the block device is immediately available at boot time, without requiring any intervention and scanning from the operating system. Not having to manage bad blocks and wear leveling also keeps software simpler, of course at the cost of less control as we said. Some board makers, for example the engineers at CALAO Systems, even predict the extinction of raw flash in the next years. Raw flash may just be kept for specific industrial applications, but would then get very expensive because of low production volume.
Fortunately, we are not completely stuck with no clue about the internals of such flash devices. Arnd Bergmann has studied cheap flash media and has developed flashbench, a benchmarking tool to find their erase block size. This allows to optimize file system settings and get huge performance boosts on these flash media, and reduce the number of block erases. Arnd has described is work in a very interesting article on LWN.net.
Other than that, you are still stuck with an opaque wear leveling mechanism, and it’s always wise to use techniques to minimize the number of writes:
Do not put a swap area on flash storage
Whenever possible, mount your filesystems as read-only, or use read-only filesystems (SquashFS)
Keep volatile files such as log files and locks in RAM (tmpfs). You do not need to keep them across reboots anyway, and you do not want to create unnecessary disk activity because of them.
Conclusions and what to remember
If you develop or hack a device with raw flash, your best option is to use the JFFS2 filesystem for small partitions, with the CONFIG_SUMMARY option. For medium to very large partitions, UBIFS will be the best compromise in terms of speed, size and boot time. However, you may get slightly better performance with YAFFS2, but at the expense of size.
If you have a device with only flash storage with a block interface, for example an SD card, download flashbench from Arnd Bergmann and optimize the settings of your filesystems to get the best performance out of your storage, and optimize its lifetime.
If you reached this part of the article, you have the patience and interest required to contribute to the MTD subsystem of the Linux kernel. Contributions, code reviews and new ideas are welcome!
Our partner ISEE is famous for their IGEPv2 board that we use in our embedded Linux course. This board is both powerful (running at 1 GHz) and featureful (on-board WiFi and Bluetooth, many connectors and expansion capabilities).
The good news is that ISEE has started to develop a new IGEPv5 board, which will be based on the new OMAP5 processor from Texas Instruments. This processor features in particular 2 ARM Cortex A15 cores running at up to 2 GHz, DDR3 RAM support, USB3, full HD 3D recording, and supporting 4 displays and cameras at the same time. Can you imagine what systems you could create with such a CPU?
If you are interested in such a board, it is still time for you to give them your inputs and expectations.
What should a perfect OMAP5 board be like? Don’t hesitate to leave your comments on this blog post. Be sure that ISEE will pay attention to them.
We are just returning from Barcelona, Spain, after participating to the 2012 edition of the Embedded Linux Conference Europe. My colleague Thomas Petazzoni has delivered the below presentation:
Your New ARM SoC Linux Support Check-List
Since Linus Torvalds raised warnings about the state of the ARM architecture support in the Linux kernel, a huge amount of effort and reorganization has happened in the way Linux supports ARM SoCs. From the addition of the device tree to the pinctrl subsystem, from the new clock framework to the new rules in code organization and design, the changes have been significant over the last one and half year in theARM Linux kernel world.
Based on the speaker’s experience on getting the new Marvell Armada 370 and Armada XP SoC supported in the mainline Linux kernel, we will give an overview of those changes and summarize the new rules for ARM Linux support. We aim at helping developers willing to add suppot for new ARM SoCs in the Linux kernel by providing a check-list of things to do.
Thomas Petazzoni is an embedded Linux engineer and trainer at Bootlin since 2008. He has been involved with multiple projects around the Linux kernel, especially the mainlining of Marvell Armada 370/XP SoCs support. He is also a major contributor to the Buildroot embedded Linux build system with more than 1100 patches merged.
The presentation slides and their sources are now available here. We have also shot a video of Thomas’ talk and it should be available in the next weeks. Stay tuned!
Do not hesitate to contact us if you are looking for engineers to port Linux to new hardware.
After months of preparation, and after completing our first Android system development training session, we are proud to release our complete training materials under the Creative Commons Attribution – Share Alike 3.0 license.
Our course targets engineers who need to develop embedded systems with Google Android, customizing Android to support specific hardware and product requirements.
Here are the main topics that we cover:
Introduction to Android
Android source code and compiling
Linux kernel, configuration, compiling and booting
Bootloaders for Android
Android changes to the Linux kernel
Supporting new hardware
Development and debugging with ADB
Android’s build system
The Android filesystem
Android native layer and calling a C program from Android
Just like the materials for our Embedded Linux system development and Embedded Linux kernel and driver development courses, this new courseware is meant to be be a useful resource for product developers around the world. It could also be used by engineers with sufficient Android experience to train their colleagues without having to spend months preparing slides and practical labs as we did.
Practical labs are conducted on the Devkit8000 embedded board from Embest, with a TI OMAP3 CPU and an LCD touchscreen. As we use Linaro Android in our labs, it should be relatively easy to make these labs on other development boards with a Linaro supported CPU (TI OMAP, Freescale i.MX53 and i.MX6, ST Ericsson Nova and Samsung Exynos).
LaTeX sources for these materials are available on Bootlin’spublic git server.
To build these documents from source, you may need to install several packages (thanks to Huy, Bui Quang, for contributing the package list):
git clone git://github.com/bootlin/training-materials.git
cd training-materials
make full-android-labs.pdf
make full-android-slides.pdf
The LaTeX format and the git repository make it easy to identify changes made since the last time you accessed the materials. We even have a training-materials-updates mailing list to receive notifications for updates to all our training materials. Compared to the Open Document Format that we used in the past, this should also make it considerably simpler to translate these documents and keep translations up to date.
As usual, comments, suggestions and translations are welcome. You can write to the feedback address available on our contact page.
In a previous post, we detailed all the talks of the Embedded Systems and Open Hardware track of the Libre Software Meeting, taking place in Geneva in early July.
Bootlin will have a quite important presence at this event, with three talks and one tutorial given by Bootlin engineers. You’ll find below the descriptions of the talks given by Bootlin. Both my colleague Maxime Ripard and myself will be present at Libre Software Meeting, and we will be happy to meet you there to discuss Embedded Linux and Android topics!
A look through the Android Stack
Android has established itself in the past years as a major player in the mobile market, outperforming any other mobile systems.
To do so, Google relied both on well established open-source components, such as the Linux Kernel, and munching them together in a brand new userspace environment. This talk will detail the most important components of Android userspace and the interactions between them that allow developers to face a consistent API for their applications.
This talk will be given on Tuesday 9th July 2012, at 14:00, by Maxime Ripard, embedded Linux and Android engineer at Bootlin. Maxime is also teaching our newest training course on Android system development.
Buildroot: a nice, simple and efficient embedded Linux build system
Started in late 2001 by uClibc developers, Buildroot has grown over its 10 years history from a testing tool for the uClibc C library to a complete, vendor-neutral, embedded Linux build system. Until early 2009, the project was mostly unmaintained and the quality slowly decreased, frustrating many Buildroot users. Fortunately, since early 2009, Peter Korsgaard took over the maintainership of Buildroot, and the project has considerably evolved since then: stable releases are published every three months, the user and developer community has grown significantly, the existing features have been cleaned up, many other new features have been added, the project is no longer uClibc-specific and the quality has been vastly improved. Buildroot now offers a nice, simple and efficient mechanism to build small to medium sized embedded Linux systems, such as the ones found in many industrial systems or highly dedicated systems. Many users are amazed about how easy it is to get started with Buildroot, especially compared to other build systems. This presentation will show how Buildroot can be used to build embedded Linux systems, highlighting the new features and improvements made over the last few years, and detailing how the simplicity of Buildroot allows you to focus on developing the applications for your system. A quick overview of the future Buildroot developments will also be provided.
This talk will take place on Wednesday 10th July at 17:00 and will be given by Thomas Petazzoni, embedded Linux engineer at Bootlin, and long time Buildroot contributor.
Linux kernel on ARM: consolidation work
In Spring 2011, Linus Torvalds asked the ARM Linux maintainers to clean up the contents of arch/arm/ in the Linux kernel code by doing more consolidation between ARM sub-architectures.
More than a year later, a lot of work has been accomplished in this area, especially thanks to the introduction of the device tree for the ARM architecture, the pinctrl subsystem and the clock framework into the Linux kernel.
Through this talk, we will present the challenges the ARM architecture creates in terms of Linux kernel support, and then describe from a technical point of view how the device tree, the pinctrl subsystem and the clock subsystem work and how they can improve the consolidation between different ARM sub-architectures.
The talk will be designed to be accessible to an audience having only a moderate knowledge of kernel programming and internals, and will therefore provide enough context for such audience to understand the issues that those different mechanisms are striving to solve.
This talk will take place on Thursday 11th July at 10:00 and will be given by Thomas Petazzoni, embedded Linux engineer at Bootlin.
Tutorial on using Buildroot, a nice, simple and efficient embedded Linux build system
Started in late 2001 by uClibc developers, Buildroot has grown over its 10 years history from a testing tool for the uClibc C library to a complete, vendor-neutral, embedded Linux build system. Until early 2009, the project was mostly unmaintained and the quality slowly decreased, frustrating many Buildroot users. Fortunately, since early 2009, Peter Korsgaard took over the maintainership of Buildroot, and the project has considerably evolved since then: stable releases are published every three months, the user and developer community has grown significantly, the existing features have been cleaned up, many other new features have been added, the project is no longer uClibc-specific and the quality has been vastly improved. Buildroot now offers a nice, simple and efficient mechanism to build small to medium sized embedded Linux systems, such as the ones found in many industrial systems or highly dedicated systems. Many users are amazed about how easy it is to get started with Buildroot, especially compared to other build systems.
This workshop follows the Buildroot presentation proposed in the same topic. During one half-day participants will be introduced on how to efficiently use Buildroot for their own projects:
Basic usage of Buildroot: generate the first system, boot it on a hardware platform
Add packages to Buildroot
Customize Buildroot for real-life projects: how to integrate project specific patches, configuration and customization
Participants are invited to come with their own laptop, installed with a sufficiently recent GNU/Linux distribution. Participants are recommended to attend the Buildroot talk by the same speaker before attending the workshop, as the talk will give an overall introduction on Buildroot.
This tutorial will take place on Thursday 11th July from 14:00 to 17:00 and will be given by Thomas Petazzoni, embedded Linux engineer at Bootlin, and long time Buildroot contributor.
The Libre Software Meeting is a community-driven free software event that exists since 2000, composed of talks and workshops. Its 2012 edition will take place from July 7th to July 12th in Geneva, Switzerland.
In the context of this conference, I was responsible with Florian Fainelli from the OpenWRT project to organize the Embedded systems and open hardware track. This track will offer an interesting selection of talks related to embedded topics, concentrated between July 9th and July 11th:
Make a free camera fly with free hardware, Éric Boudrand. It will be the only talk of the track that will take place in French. The rest of the track contents will be in English.
The real time page I wrote for Atmel was finally released on the Linux4Sam Atmel Wiki. The purpose of this page was to help new comers to use real time features with Atmel CPUs and to present the state of the real time support.
Here are some figures associated to this work:
On this page I present the results of more than 300 hours of benchmarks!
During the setup and the tuning tests ran for more than 600 hours.
Analysis and formatting took a few dozen hours of work.
The benchmarks have been run on 3 boards, 3 flavors of Linux (vanilla, PREEMPT-RT patches, Xenomai co-kernel approach), and 2 kinds of tests (timer-based and GPIO-based)
I recently had the opportunity to use Yocto. I already practiced quite a lot with OpenEmbedded before. You can see Yocto as a project derived from OpenEmbedded even it is a bit more than that.
In fact, Yocto is made of Poky (a build system based on OpenEmbedded), a few added build tools (swabber, pseudo, etc.), as well as a set of meta data allowing to create embedded distributions for a number of targets.
The strength but also the weakness of OpenEmbedded is that it a very flexible build system. It can make production root filesystems, but also a complete distribution with its ready to use package repository, and this for multiple hardware platforms. It makes it a difficult system to get started and get efficient with. Even two years ago, the OpenEmbedded documentation contributed to making it difficult to get started. Indeed, OpenEmbedded did supply some documentation, but which only started to make sense once you start mastering it. This is quite a paradox for a piece of documentation. It lacked the elements allowing developers to understand its operation well.
With Yocto, I was pleased to realize that substantial progress had been made on this side. The project comes with documentation that is much more exhaustive and above all much more accessible for beginners. Getting started with it is still not completely straightforward, but this time, this is rather because of the complexity and the rich features of the tool.
In a few hours, I managed to develop a minimalistic BSP (Board Support Package) for a given board (in this case a AT91SAM9G20-EK). The concept of layer allows to have a configuration layer specific to a given piece of hardware. You can even support multiple hardware platforms at once and add specific packages. A layer is indeed just a set of packages and configurations (or configuration overrides). The BSP is just a layer specific to one or several pieces of hardware.
As you can see, even to support a simple embedded board, there is already a number of concepts to deal with. There are also multiple ways of achieving the same result but which will be easier or more difficult to maintain. The concept of “BSP” for Yocto is therefore a kind of guideline to allow the Yocto community to have a common point of reference. I will try to illustrate the use of a BSP on the AT91SAMG20-EK board here and/or on my Google+ page.
Another significant progress is optimizing build time for a “minimalistic” target, which went down from more than three hours to just over one hour now. It remains a long time for a very simple target.
To build a filesystem image with only a few components, Buildroot remains much more appropriate. For systems that require a great number of components, more advanced functionality is then often needed, such as managing a package repository and supporting multiple hardware platforms at once for example. In this case, Yocto remains the best (the only?) option, all the more as this project addresses the traditional weaknesses of OpenEmbedded.
The 2012 edition of the Embedded Linux Conference took place on February 15-17th 2012 at Redwood Shores near San Francisco in California. Three engineers of Bootlin attended this conference, and we reported every day our impressions about the talks, see our blog posts for day 1, day 2 and day 3. We have now taken the time to encode all the videos we have recorded during this event, and are proud to distribute them today.
It is worth noting that for the first time, the Linux Foundation was also recording videos of the talks, the Linux Foundation videos are available from video.linux.com, and we included links to these videos below for the different talks.
We hope that those of you who couldn’t attend the conference will enjoy those videos, with many great talks on technical embedded Linux topics.
Loïc Pallardy Saving the Power Consumption of the Unused Memory Slides Bootlin video (46 minutes): full HD (378M), 450×800 (125M)
Bernhard Rosenkränzer Linaro What Android and Embedded Linux Can Learn From Each Other Slides Linux Foundation video Bootlin video (40 minutes): full HD (370M), 450×800 (129M)
Ricardo Salveti de Araujo Linaro Ubuntu on ARM: Improvements and Optimizations Done By Linaro Slides Linux Foundation video Bootlin video (46 minutes): full HD (301M), 450×800 (140M)
Sean Hudson Mentor Graphics, Inc. Embedded Linux Pitfalls Slides Bootlin video (51 minutes): full HD (483M), 450×800 (176M)
Vincent Guittot Linaro Comparing Power Saving Techniques For Multicore ARM Platforms Slides Linux Foundation video Bootlin video (57 minutes): full HD (307M), 450×800 (154M)
Bruce Ashfield Wind River A View From the Trenches: Embedded Functionality and How It Impacts Multi-Arch Kernel Maintenance Slides Bootlin video (54 minutes): full HD (741M), 450×800 (222M)
Matt Porter Texas Instruments Passing Time With SPI Framebuffer Driver Slides
Bootlin video (54 minutes): full HD (565M), 450×800 (172M)
Wookey Linaro Multiarch and Why You Should Care: Running, Installing and Crossbuilding With Multiple Architectures Slides Bootlin video (42 minutes): full HD (453M), 450×800 (143M)
Amit Daniel Kachhap Linaro/Samsung A New Simplified Thermal Framework For ARM Platforms Slides Linux Foundation video Bootlin video (41 minutes): full HD (226M), 450×800 (115M)
Tsugikazu Shibata NEC On The Road: To Provide the Long-Term Stable Linux For The Industry Slides Linux Foundation video Bootlin video (32 minutes): full HD (304M), 450×800 (95M)
Thomas P. Abraham Samsung Electronics Experiences With Device Tree Support Development For ARM-Based SOC’s Slides Bootlin video (44 minutes): full HD (509M), 450×800 (155M)
Thomas Petazzoni Bootlin Buildroot: A Nice, Simple, and Efficient Embedded Linux Build System Slides Linux Foundation video Bootlin video (56 minutes): full HD (594M), 450×800 (182M)
Edward Hervey Collabora GStreamer 1.0: No Longer Compromise Flexibility For Performance Slides Linux Foundation video Bootlin video (49 minutes): full HD (540M), 450×800 (174M)
Tim Bird Sony Network Entertainment Embedded-Appropriate Crash Handling in Linux Slides Linux Foundation video Bootlin video (49 minutes): full HD (292M), 450×800 (142M)
Mark Gisi Wind River Systems The Power of SPDX – Sharing Critical Licensing Information Within a Linux Device Supply Chain Linux Foundation video Bootlin video (49 minutes): full HD (498M), 450×800 (164M)
Yoshitake Kobayashi Toshiba Ineffective and Effective Ways To Find Out Latency Bottlenecks With Ftrace Slides Linux Foundation video Bootlin video (37 minutes): full HD (251M), 450×800 (108M)
Ohad Ben-Cohen Wizery / Texas Instruments Using virtio to Talk With Remote Processors Slides Linux Foundation video Bootlin video (54 minutes): full HD (582M), 450×800 (182M)
Elizabeth Flanagan Intel Embedded License Compliance Patterns and Antipatterns Linux Foundation video Bootlin video (44 minutes): full HD (391M), 450×800 (144M)
Frank Rowand Sony Network Entertainment Real Time (BoFs) Slides Bootlin video (77 minutes): full HD (924M), 450×800 (288M)
Mike Turquette Texas Instruments Common Clock Framework (BoFs) Slides Bootlin video (53 minutes): full HD (333M), 450×800 (148M)
Hunyue Yau HY Research LLC Userland Tools and Techniques For Linux Board Bring-Up and Systems Integration Slides Linux Foundation video Bootlin video (51 minutes): full HD (407M), 450×800 (136M)
Koen Kooi The Angstrom Distribution Producing the Beaglebone and Supporting It Linux Foundation video Bootlin video (42 minutes): full HD (398M), 450×800 (126M)
Flash storage, also called solid state, has multiple advantages over rotating storage. First, the absence of mechanical and moving parts eliminate noise, increase reliability and resistance to shock and vibrations, and also reduces heat dissipation as well as power consumption. Second, random access to data is also much faster, as you no longer have to move a disk head to the right location on the medium, which can take milliseconds. 
After months of preparation, and after completing our first Android system development training session, we are proud to release our complete 
Practical labs are conducted on the 
