Linux 2.6.33 features for embedded systems

Interesting features for embedded Linux system developers

Penguin workerLinux 2.6.33 was out on Feb. 24, 2010, and to incite you to try this new kernel in your embedded Linux products, here are features you could be interested in.

The first news is the availability of the LZO algorithm for kernel and initramfs compression. Linux 2.6.30 already introduced LZMA and BZIP2 compression options, which could significantly reduce the size of the kernel and initramfs images, but at the cost of much increased decompression time. LZO compression is a nice alternative. Though its compression rate is not as good as that of ZLIB (10 to 15% larger files), decompression time is much faster than with other algorithms. See our benchmarks. We reduced boot time by 200 ms on our at91 arm system, and the savings could even increase with bigger kernels.

This feature was implemented by my colleague Albin Tonnerre. It is currently available on x86 and arm (commit, commit, commit, commit), and according to Russell King, the arm maintainer, it should become the default compression option on this platform. This compressor can also be used on mips, thanks to Wu Zhangjin (commit).

For systems lacking RAM resources, a new useful feature is Compcache, which allows to swap application memory to a compressed cache in RAM. In practise, this technique increases the amount of RAM that applications can use. This could allow your embedded system or your netbook to run applications or environments it couldn’t execute before. This technique can also be a worthy alternative to on-disk swap in servers or desktops which do need a swap partition, as access performance is much improved. See this LWN.net article for details.

This new kernel also carries lots of improvements on embedded platforms, especially on the popular TI OMAP platform. In particular, we noticed early support to the IGEPv2 board, a very attractive platform based on the TI OMAP 3530 processor, much better than the Beagle Board for a very similar price. We have started to use it in customer projects, and we hope to contribute to its full support in the mainline kernel.

Another interesting feature of Linux 2.6.33 is the improvements in the capabilities of the perf tool. In particular, perf probe allows to insert Kprobes probes through the command line. Instead of SystemTap, which relied on kernel modules, perf probe now relies on a sysfs interface to pass probes to the kernel. This means that you no longer need a compiler and kernel headers to produce your probes. This made it difficult to port SystemTap to embedded platforms. The arm architecture doesn’t have performance counters in the mainline kernel yet (other architectures do), but patches are available. This carries the promise to be able to use probe tools like SystemTap at last on embedded architectures, all the more if SystemTap gets ported to this new infrastructure.

Other noticeable improvements in this release are the ability to mount ext3 and ext2 filesystems with just an ext4 driver, a lightweight RCU implementation, as well as the ability to change the default blinking cursor that is shown at boot time.

Unfortunately, each kernel release doesn’t only carry good news. Android patches got dropped from this release, because of a lack of interest from Google to maintain them. These are sad news and a threat for Android users who may end up without the ability to use newer kernel features and releases. Let’s hope that Google will once more realize the value of converging with the mainline Linux community. I hope that key contributors that this company employs (Andrew Morton in particular) will help to solve this issue.

As usual, this was just a selection. You will probably find many other interesting features on the Linux Changes page for Linux 2.6.33.

Buildroot 2010.02 released, contributions from Bootlin!

Buildroot logoBuildroot is a embedded Linux system build system. It automates the process of downloading, configuring, compiling and installating all the components of an embedded Linux system, from Busybox to more complicated software stacks using Gtk, Qt, X.org, Gstreamer, etc. Buildroot is easy to use and extend, making it a nice choice for small to medium-sized embedded Linux systems.

As promised by the fixed-release schedule, a 2010.02 release has been published on Friday, with numerous improvements over the previous version 2009.11, many of which are part of the general cleanup process that the project is doing since the beginning of 2009. These improvements are detailed in the project CHANGES file.

Thomas Petazzoni, from Bootlin, implemented several of these improvements :

  • Creation of a package infrastructure for non-autotools packages. Buildroot had for a long time an infrastructure to factorize the code needed to build packages based on the autotools build systems. But all other packages were using hand-made Makefiles, which were hard to write and generated a lot of code duplication. Therefore, we have introduced an infrastructure that makes adding new packages much easier, and which allows us to cleanup the existing codebase significantly by factorizing a lot of common code. The autotools infrastructure has also been reworked on top of the generic infrastructure to avoid code duplication as well. At the same time, we have significantly improved the documentation on how to add new packages. This infrastructure is a building block that will allow us to easily add more features to all packages in Buildroot (such as package generation).
  • Removal of the external toolchain source mechanism, which was merged with the normal toolchain building procedure. This special casing was implemented to allow the compilation of AVR32 toolchains, but such an additional complexity wasn’t needed. Now, Buildroot continues to build AVR32 toolchains as it used to do, but the code is much cleaner. Another illustration of our large cleanup effort.
  • Many, many, many fixes to different packages, many of them to ensure that we do not depend on development packages being installed on the host. This is very important to ensure that our build procedure is as independent as possible from the development machine configuration.

From the list of contributors, ordered by the number of patches, Thomas Petazzoni of Bootlin has been the first Buildroot contributor for this last release :

$ git shortlog -e -n -s 2009.11..2010.02  (removed e-mail addresses)
139 Thomas Petazzoni
124 Peter Korsgaard
26  Lionel Landwerlin
23  Gustavo Zacarias
7   Julien Boibessot
4   Nigel Kukard
4   Sven Neumann
2   Anders Darander
2   Chris Packham
2   Daniel Mack
2   H Hartley Sweeten
2   Richard van Paasen
2   Will Wagner
2   William Wagner
2   Yann E. MORIN
1   Cameron Hutchison
1   Clark Rawlins
1   Francisco Gonzalez
1   Francisco Gonzalez Morell
1   Hans-Christian Egtvedt
1   Lionel Landwerlin
1   Ormund Williams
1   Rob Alley
1   Sagaert Johan
1   grante

For the next release, we will work on additional cleanup of Buildroot and particularly the target/ directory, which contains the code to build the Linux kernel, different bootloaders, and to generate the final root filesystem image in various formats. Improving support for external toolchains is also on our TODO list : supporting multilib toolchains such as the CodeSoucery toolchain, and fixing a long-standing issue with libtool.

Don’t hesitate to try Buildroot, and to report your successes and failures on the mailing-list, in our bug tracker, or on our IRC channel, #uclibc on Freenode.

Tourist suggestions in Nice

NiceTourist tips for participants to our training sessions in Nice, France.

We used to organize public training sessions about embedded Linux and kernel and drivers training sessions in Vence, near nice. The sessions are now run in Avignon, but we can still organize sessions for customers in the area, or anywhere in the world.

To explore the area, we recommend to rent a car. Cheap public transportation is available, but their service doesn’t go everywhere, and above all, doesn’t respect its schedules well. You should leave your car though, if you wish to visit the centers of Nice or Cannes. Traffic and parking rates are horrible, and a good solution is to park at a train station, and reach the center of these cities by local train.

AntibesOf course, these tips only reflect my own interests and tastes. There are certainly others other worthy things to do and see in the area. I would recommend to buy a tourist guide book about the French Riviera, which will have a much more exhaustive and objective coverage. See also the French Riviera tourism website.

Around Vence

GourdonVence itself, where our sessions took place, is a nice medieval old town with very nice pedestrian streets. It takes just 5 minutes to reach the old town by foot from the hotel.

In the area, you mustn’t miss Saint Paul de Vence, a beautiful medieval village 3 kilometers away, which is famous worldwide. It is very nice to visit, though you may be disappointed by its lack of real life there. There are lots of foreign tourists and only art galleries and restaurants. If you look for similar but more authentic villages in the area, you can also visit Gourdon, Tourrettes sur Loup or Valbonne.

You shouldn’t miss Antibes either. It is the only well preserved old town on the sea shore, like the old villages uphill, Unlike Nice, and above all Monaco and Cannes, it’s an authentic place, with people like you and me, and less display of money and luxury.

Visiting cities: Nice, Cannes, Monaco

I would suggest to visit Nice and Monaco, and skip Cannes, which doesn’t really have anything special the others don’t have. Nice has a nice old town, and a beautiful park above. You can also enjoy walking or roller skating on the beach front (La Promenade des Anglais).

I don’t find Monaco as attractive as it is famous. The old town and the castle are nice, but the whole area is very dense with lots of tall, ugly concrete buildings. One thing that you shouldn’t miss there is its amazing oceanographic museum.

EzeI would still recommend to visit Monaco if you have enough time though, not for the place itself, but for the road between Nice and itself. You can go to Monaco through the Moyenne Corniche, a road on a very steep cliff and breathtaking views over the Mediterranean. A few James Bond movie scenes where shot there by the way, and the place should look familiar to you. On your way, you must stop at Èze, a beautiful village 1,200 feet above the sea. On the way back, you can take the Grande Corniche which offers a very nice scenery as well.

Exploring natural resources

Even if you don’t have a car, you can hike up a hill on top of Vence, called the Baou des Blancs. Ask the hotel people for where the trail starts. On top of it, you will discover a fantastic view on the whole “Bay of Angels”. Depending on how fast you walk, you may need from 3 to 4 hours from Vence for the whole hike. If you have a car or take a taxi to get you to the beginning of the trail, 90 minutes may be sufficient.

Cinque TerreIf you have a car, good hiking shoes and enough time, you could visit the neighboring Alp mountains, locally culminating at more than 9,000 feet. In particular, you could go to the Valley of Marvels.

With the same equipment, you could also explore the Verdon Gorge, a spectacular canyon 90 minutes away from Nice.

If you have at least 2 days available, you could also go to the Cinque Terre in Italy, 3 hours away driving. You can also reach the place by train, though you may have 1 or 2 train connections on your way. Just look at the pictures.

You should also be interested in visiting Provence, within 1 to 3 hours of driving. This is one of the most celebrated parts of France, with beautiful scenery, monuments and a special “art de vivre”. There are so many things to do and see there that its worth buying a special book about it.

CorsicaLast but not least, if you have at least 4 to 5 days, what about going to Corsica? You can reach this island by ferry boat (3 to 4 hours from Nice) or by plane. This is one of the most beautiful places on earth, with an amazing varianty of breathtaking landscapes: mountains, beaches, rivers. You won’t forget your stay.

Embedded Linux practical labs with the Beagle Board

Note: the materials for training with the Beagle Board are no longer available, and would be significantly out of date anyway. We advise you to check our Embedded Linux System Development and Linux Kernel and Driver Development training courses for up-to-date instructions that work on cheaper boards, which are still available on the market today. And if you still have an old Beagle board, it will be an interesting exercise to adapt our current labs to run them on such hardware.

We were asked to customize our embedded Linux training session with specific labs on OMAP 3530 hardware. After a successful delivery on the customer site, using Beagle boards, here are our training materials, released as usual under the terms of the Creative Commons Attribution-ShareAlike 3.0 license:

If you are the happy owner of such a board (both attractive and cheap), or are interested in getting one, you can get valuable embedded Linux experience by reading our lecture materials and by taking our practical labs.

Here’s what you would practise with if you decide to take our labs:

  • Build a cross-compiling toolchain with crosstool-NG
  • Compile U-boot and the X-loader and install it on MMC and flash storage.
  • Manipulate Linux kernel sources and apply source patches
  • Configure, compile and boot a Linux kernel for an emulated PC target
  • Configure, cross-compile and boot a Linux kernel on your Beagle Board
  • Build a tiny filesystem from scratch, based on BusyBox, and with a web server interface. Practice with NFS booting.
  • Put your filesystem on MMC storage, replacing NFS. Practice with SquashFS.
  • Put your filesystem on internal NAND flash storage. Practice with JFFS2 too.
  • Manually cross-compile libraries (zlib, libpng, libjpeg, FreeType and DirectFB) and a DirectFB examples, getting familiar with the tricks required to cross-compile components provided by the community.
  • Build the same kind of graphical system automatically with Buildroot.
  • Compile your own application against existing libraries. Debug a target application with strace, ltrace and gdbserver running on the target.
  • Do experiments with the rt-preempt patches. Measure scheduling latency improvements.
  • Implement hotplugging with mdev, BusyBox’s lightweight alternative to udev.

Note that the labs were tested with Rev. C boards, but are also supposed to work fine with Rev. B ones. You may also be able to reuse some of our instructions with other boards with a TI OMAP3 processor.

Of course, if you like the materials, you can also ask your company to order such a training session from us. We will be delighted to come to your place and spend time with you and your colleagues.

LZO kernel compression

As Michael stated in his review of the interesting features in Linux 2.6.30, new compression options have been recently added to the kernel. We therefore decided to have a look at those compression methods, from a compression ratio and decompression speed point of view.

This comparison will be based on “self-extractible kernels”, that is, kernel images containing bootstrap code allowing them to extract a compressed image. As underlined in the previous article, this approach is not used on all architectures. Blackfin, notably, chose a different path and compresses the whole kernel image, without including bootstrap code. While this has the clear advantage of making compression much simpler with respect to kernel code, it forces decompression out to the bootloader code.

Each of those methods has its advantages. Indeed, the Blackfin approach relies on the bootloader to provide the necessary functions, so that may be a problem to do things like bypassing u-boot to reduce the boot time. On the other hand, implementing it only once in the bootloader (as architecture-independent code) makes it unnecessary to write the low-level bootstrap code for each architecture in the kernel, which is surely interesting on virtually all architectures, the notable exceptions being x86 and ARM.

Gzip (also known as Zlib or inflate) has been the traditional (and, as a matter of fact, only) method used to compress kernels. Consequently, we’ll use it as the reference in the following tests. Our test environment is as follows:

ARM9 AT91SAM9263 CPU, 200MHz, using the mainline arch/arm/configs/usb-a9263.config.

This comparison includes figures for LZO, a new kernel image compression method that I have contributed to the Linux sources, and which hopefully will make its way into the mainline kernel. LZO support in the kernel is only new for kernel decompression, as it is already used by JFFS2 and UBIFS. LZO is a stream-oriented algorithm, and although its compression ratio is lower than that of gzip, decompression is lightning-fast, as we will soon find out.

So, here are the figures, average on 20 boots with each compression method:

Uncompressed 3.24Mo – – 200%
LZO 1.76Mo 0.552s 70% 109%
Gzip 1.62Mo 0.775s 100% 100%
LZMA 1.22Mo 5.011s 646% 75%

Bzip2 has not been tested here: the low-level bootstrap file, head.S, only allocates 64Kb for use by malloc() on ARM. Some quick tests showed that the kernel would not extract with less than 3.5Mib of malloc() space. That would require to modify head.S so that malloc can use more memory, which we will not do here. However, given that enough memory is usable on the system, one could well use bzip2. All the other algorithms performed the extraction using the standard 64Kib malloc space.

From the results, we can clearly see that LZMA is nearly unsuitable for our system, and should be considered only if the space constraints for storing the kernel are so tight that we can’t afford to use more space that was is strictly necessary.

LZO looks like a good candidate when it comes to speeding up the boot process, at the expense of some (almost neglectable) extra space. Gzip is close to LZO when it comes to size, although extraction is not as fast. That means that unless you’re hitting corner cases, like only having enough space for a Gzip compressed image but not for one made with LZO, choosing the latter is probably a safe bet.

Besides, the LZO-compressed kernel size is about 54% the size of the uncompressed kernel. As the kernel load time varies linearly with its size, load time for an uncompressed kernel doubles. While 0.55s are won because there’s no need to run a decompression algorithm, you spend twice as much time loading the kernel. This time is not negligible at all compared to the decompression time. Indeed, loading the uncompressed image takes roughly 0.8s. That means that at the cost of slowing down the boot process by 0.15s (compared to an uncompressed kernel), one gets a kernel image which is roughly twice as small. Rather nice, isn’t it?

Bootlin at ELCE 2009

Grenoble

As usual, we won’t miss this year’s edition of the Embedded Linux Conference Europe, which has always been a great source of information and encounters for embedded Linux developers.

Here are details about our involvement this year.

  • I am part of the organization committee, in particular the coordinator for the Technical Showcase.
  • Taking advantage of his stay in Grenoble, my colleague Thomas Petazzoni will make an embedded Linux presentation on Tuesday, Oct. 13 at 7:30 pm, at GUILDE, the local Linux user group.
  • Thomas and I will be present at the Embedded Systems Exhibition on Wednesday, Oct. 14, sharing a booth with our partner CALAO Systems. The exhibition entry is free of charge, and this will be an excellent opportunity to meet us and have enough time to talk about your topics of interest.
  • Thomas will lead the Buildroot BOF with Peter Korsgaard, Buildroot’s maintainer, at 5:35 pm on Thursday, Oct. 15. This informal session will allow users and developers to meet and exchange ideas.
  • I will be the leader of the Small Business BOF on Thursday 15 at 6:35 pm, an informal session for small embedded Linux companies interested in sharing experience and best practices, and of course to know each other better.
  • I will make a presentation on boot time reduction techniques, at 3:40 on Friday, Oct. 16.
  • Albin Tonnerre, who was an intern at Bootlin this summer, will participate to the Technical Showcase at 12:00 am on Friday, Oct. 16, showing the benefits of LZO decompression on kernel boot time. During his internship, Albin made very nice contributions to boot time reduction, power management on AT91 and to U-boot board support.
  • Thomas Petazzoni will also participate to the Technical Showcase at the same time, showing Buidroot’s new features.
  • We will videotape the conferences we go to and will release the videos later on our website.
  • Thomas organizes a Buildroot developer day on Saturday, Oct. 17, allowing developers to meet and code together. Bootlin will offer lunch to the participants, and the room will be offered by CALAO Systems. There are no more seats left for space reasons.

Hope to see you in Grenoble!

Crosstool-NG 1.5.0 released, with new features!

Crosstool-NG, the successor of Crosstool, is a tool that automates the complicated process of building a cross-compilation toolchain. It provides a nice menuconfig interface to fine tune the configuration of the toolchain, before creating the toolchain automatically by retrieving, extracting, patching, configuring, compiling and installing the different components in the right order, with the right arguments and configuration.

Yann E. Morin, the maintainer of Crosstool-NG has just announced the release of Crosstool-NG 1.5.0, with the following new features:

  • Support for gcc 4.4
  • Experimental support for canadian-cross toolchains. Canadian-cross toolchains are toolchains that are built on machine A, to run on machine B and generate code for machine C, while usual cross-compilation toolchain are built on machine A, to run on machine A and generate code for machine B
  • Experimental support for AVR32, with support for operation as MMU-less, thanks to the newlib C library

In addition to these important features, bugs have been fixed and improvements have been made. Yann also switched the development of Crosstool-NG from Subversion to Mercurial, in order to ease community participation in the improvement of Crosstool-NG.

By the way, if you are interested in Crosstool-NG, don’t miss Yann E. Morin’s talk at the Embedded Linux Conference Europe 2009. See the program for details.

CALAO boards supported in mainline U-Boot

I’m happy to announce that a couple days ago, support for the CALAO SBC35-A9G20, TNY-A9260 and TNY-A9G20 boards made its way into the U-Boot git repository. Sadly, it’s not possible to boot from an MMC/SD card with the SBC35 yet, but it’s something I’m currently working on.

Support for all these cards will be available in the next U-Boot release, due in November.

Buildroot simplified for users!

Buildroot logoYesterday, a set of patches I’ve authored that aims at simplifying Buildroot for users has been merged into the official version of the project, and will therefore be part of the next stable release (scheduled for November, according to our 3 months release cycle). This work is probably my major contribution to Buildroot, outside of external toolchain support and various fixes here and there. Here are quick details about the improvements brought by these patches :

  • Remove the “project” feature. The project feature removal was the main point of this patch set. This feature, that allows to compile a system for different, but very similar platforms, without recompiling everything from scratch, was rarely used and introduced a lot of complexity in the usage of Buildroot for newcomers. Who hasn’t been confused by this project_build_arm directory? This thing is gone now.
  • Remove the BOARD/LOCAL feature, which duplicates another way of adding support for new targets in Buildroot. This is the kind of feature that has been added at the time Buildroot was basically unmaintained, when nobody was able to say « Hey, but you’re just trying to-reimplement something that already exists »
  • Move all output directories in an output directory. By default, when Buildroot is compiled, it generates several directories in the middle of its source code. Now, with this patch, everything is grouped into an output directory, unless out-of-tree compilation is used, of course (with O=)
  • Remove TOPDIR_PREFIX and TOPDIR_SUFFIX since the same effect could already be done using out-of-tree compilation with O=. Another duplicated feature that should never have reached the tree.
  • Rename the output directories. Now that everything is properly stored in an output directory, it was time to rename the subdirectories to make them more meaningful. So now, we have build where all packages are built, images that contains the final binary images of the root filesystem and the kernel, staging which contains the staging directory (all packages installed with their development headers and libraries), target that contains the root filesystem for the target (without the device files), host that contains the installation of the host tools that Buildroot requires for its execution, stamps that contains the stamp files used by Buildroot to keep track of the compilation progress. Therefore, all directories such as build_ARCH or toolchain_build_ARCH have disappeared.
  • Major documentation update, to of course make sure that our documentation is up-to-date with the latest changes.

Getting all these changes mainlined is really a nice thing. I also have tons of other ideas to improve Buildroot infrastructure, and I’m sure the coming Buildroot Developer Day will be a great opportunity to discuss these.