Kernel 2.6.28 is out with a few Bootlin contributions

A few hours before Christmas, Linus Torvalds released the latest stable version of the Linux kernel, 2.6.28. Jake Edge from LWN sums up the major highlights of this new release: « Some of the highlights of this kernel are the addition of the GEM GPU memory manager, the ext4 filesystem is no longer “experimental”, scalability improvements in memory management via the reworked vmap() and pageout scalability patches, moving the -staging drivers into the mainline, and much more ». As usual, the Kernel Newbies website offers an excellent human-readable summary of the changes.

Of particular interest to embedded developers will be the new boot tracer facility, which allows to draw SVG graphs of the kernel initialization procedures execution time, in order to analyze the boot time and possibly reduce it. Of course, a lot of architecture-dependent improvements have also been made (for example OProfile support for ARMv7 CPUs but also new supported boards) and a lot of drivers have been merged or improved, as usual.

Bootlin has contributed a few patches that have been merged and released in 2.6.28. While being a small contribution compared to the 9.000+ patches added to the kernel between 2.6.27 and 2.6.28, they still slightly improve the kernel for embedded users. Part of the Linux-Tiny efforts, these patches allow to reduce the size of the kernel by disabling features that may not be necessary on embedded systems. More specifically, these patches allow :

From the existing Linux Tiny patch ideas, the only one left in the feature removal area are the multicast support removal and ethtool removal. They have already been submitted a few months ago, but got rejected by the network maintainers. I will work on them again to fix the issues and try to re-submit them later.

Finally, Jonathan Corbet has published an analysis of the 2.6.28 developement cycle in terms of contributors and changes. An interesting reading.

Bootlin at FOSDEM 2009

The Free and Open Source Developer European Meeting (FOSDEM) is a major event for open source developers in Europe. This two-days event takes place in Brussels since several years and attracts 2000-3000 people around conferences and development rooms. The program for the main tracks has been recently announced, but the program for the development rooms is not available at this time. However, I’ve been at FOSDEM the last two years and always found interesting talks and discussions.

FOSDEM Banner

Of course, I’ll be particularly interested by the Embedded Devroom, and will record videos of the talks that will be posted on Bootlin website after the conference, as usual.

If you happen to come to FOSDEM, I’ll be happy to meet you!

New LXR website

I am pleased to announce that our https://elixir.bootlin.com website is back on line.

As some of you probably noticed, our service had been down for several months. When it was working, it was based on LXR 0.9.5. This version had nice improvements over the stable release (0.3.1), like better display of the kernel sources, but on the other hand, it also had ugly drawbacks. In particular, it stored data in an SQL database. This made the server consume more CPU resources, and made it very long to index a new kernel version (about 10 hours instead of just a few minutes). Disk space was also multiplied by 3 or 4, if I recall correctly. Anyway, the major problem was that that version didn’t scale: the service was getting slower each time a new version was added. Apparently, the bigger the database, the slower the server got.

Eventually, that 0.9.5 based server just died. I didn’t change anything before this happened and everything looked all right. I checked configuration files and packages, but there seemed to be no way to make it run again. The only solution left was a brand new install from scratch.

I first evaluated the LXRng, a new fork of the software on the http://lxr.linux.no website. This new version looks nice. In particular, though it’s still using a database, this new branch seems to withstand a greater number of kernel source versions, as http://lxr.linux.no answers pretty fast now and indexes a pretty long list of versions. However, I found its interface confusing and not as convenient as it was in the original branch, especially for identifier search. It could be because this new version is not mature enough yet, or just because I was too familiar with the original interface. The best is to try by yourself!

I also tried to make a new installation of the latest CVS sources of the main branch. However, this didn’t work as expected, as I wanted to run the new service on a recent distro with long term support (Ubuntu Gutsy Gibbon). Gutsy Gibbon only supports MySQL 5.0, but LXR-CVS proved to be only compatible with MySQL 4.0. I did apply some patches, but still got SQL query errors with MySQL 5.0.

I eventually decided to go back to the good old 0.3.1 stable version, and I don’t regret it:

  • This version is extremely simple. You just need a web server with CGI scripts. No trouble with Apache2, no need to make modperl work. No need to install database software.
  • This version is extremely fast too. It just takes a few minutes to add a new version, and serving identifier searches is very fast. Just try with a widely used function, like outb. With LXR 0.9.5, it could take up to 1 minute to display all the files in which the symbol was found.
  • This version scales by design. Each supported source version has its own index files, and there is no central blob getting bigger and bigger. This simple design also makes it very easy to remove or to update source versions (like upgrading 2.6.28 sources to 2.6.28.1). With LXR 0.9.5, you had to make your own SQL queries to remove a version from the database!
  • This version lacks a few features (like direct links to C include files, or like file descriptions), but hey, the main features are there: source navigation and identifier search. The only significant feature that is kind of missing in our site is freetext search. Version 0.3.1 only supported a proprietary searching tool, so we decided to rely on Google’s search instead. This is not perfect as we won’t have version-specific search, but freetext search is a secondary feature for us anyway. We wanted to have this service back on line, with at least its main features.
  • Note that I had to make minor changes to make the website XHTML 1.0 Transitional compliant and pass the W3C Markup Validation Service checks I also fixed a bug in the diff markup script. Here is an archive of our install. Don’t hesitate to compare it with the original code and templates, and reuse our modified templates if you like them.

Thanks to this, the code hyperlinks in our kernel training slides work again at last! Every time we mention the name of a kernel source file or quote example code, you can click on the file name or on each function or structure type name, and you will be taken to the corresponding page on our LXR site!

Don’t forget that other valuable LXR websites exist for the Linux kernel. See our LXR websites list. Don’t hesitate to post a comment if you know other useful ones.

Happy New Year!

Our very best wishes for 2009!

Usually, we create special wish cards for our customers and for the whole community. Unfortunately, we didn’t have enough time last year, and this happened again this year. Actually, higher priority projects are keeping us busy:

  • Fixing our LXR website. Thanks to this, the code hyperlinks in our kernel slides work again!
  • Preparing our new training sessions. We now propose two new training agendas, one full week only about the Linux kernel, and another full week on embedded Linux system development. Last but not least, we now use real hardware in our training sessions, and not just emulated boards.
  • Processing the videos we took at the 2008 edition of the Embedded Linux Conference Europe. We hope to release them by the end of January.
  • Migrating the French part of our website to WordPress, as we did with the English one.
  • Releasing new technical documents that we haven’t had time to polish yet.
  • Making contributions to community projects (Linux kernel, QEMU, Buildroot…).
  • Working on development projects

Anyway, we really hope that this year will be very busy for you too, despite the economic slowdown. With sustainable and cost-effective solutions, backed by a huge community of developers and users, you could really make a difference.

Choosing graphical libraries for embedded systems

The free software community offers many solutions to embedded system developers willing to add graphical applications to their project. This variety of choice, typical from the free software world, has the advantage of giving several solutions, which increases the chance of finding the solution that bests suits your need, but at the same time, might confuse to choose the right one.

I made experiments with the major graphical libraries available, and reported these experiments during the Embedded Linux Conference Europe event, which took place early November 2008 in Ede, The Nederland. My presentation « Choosing graphical libraries for embedded systems » discussed DirectFB, X.org and its Kdrive variant, SDL, Nano-X, Gtk, Qt, FLTK and WxEmbedded, detailing the features, specifities, size of each solution and suitability to various use cases.

The slides are available under the Creative Commons BY-SA license : graphical-libraries.pdf (PDF), graphical-libraries.odp (Open Document Format).

While experimenting with these graphical libraries, I made a few contributions to the Buildroot project, which was used to build root filesystems including these libraries. I hope to release soon several root filesystems allowing an easy testing of these solutions, through Qemu.

uClibc 0.9.30 is available

About one year and a half after the release of the previous stable version, the release of uClibc 0.9.30 is a great event in the embedded Linux community. uClibc is a replacement for the glibc C library, implementing most of the features of glibc, while retaining a much smaller size and an incredible level of configurability.

The only changelog available is a list of Subversion commits that occurred between the 0.9.29 and the 0.9.30 releases, so it is quite difficult to extract what are the important bits. However, a news from August 2008 on uClibc.org website gives an idea of what happened in the 0.9.30 version :

  • a lot of fixes for the various architectures, and other tweaks and improvements
  • an improved configurability that allows to enable/disable a larger number of features, now including
    • Realtime-related family of SUSv functions (option UCLIBC_HAS_REALTIME, which enables aio_*() functions, mq_*() functions, mlock() family of functions, sched_*() functions, sem_*() functions, a few signal-related functions and the timer_*() functions). Threading support requires the realtime functions, so it depends on this option.
    • Advanced realtime-related family of SUSv functions (option UCLIBC_HAS_ADVANCED_REALTIME, which enables a few advanced clock_*() and mq_*() functions, and a large number of posix_spawnattr_*() and posix_spawn_*() functions)
    • epoll (option UCLIBC_HAS_EPOLL)
    • extended attributes (option UCLIBC_HAS_XATTR)
    • other options to enable/disable compatibility/deprecated APIs
  • it is now possible to build uClibc without network support at all. The global option is UCLIBC_HAS_NETWORK_SUPPORT, and can be further refined with UCLIBC_HAS_SOCKET to enable just the socket support (for example if only Unix sockets are used), UCLIBC_HAS_IPV4 to get IPv4 functionality, which of course requires the socket support, and UCLIBC_HAS_IPV6 for IPv6.

A quick look at the differences between the available options allows to see another set of features:

  • Support for the AVR32 and Xtensa architecture has been added
  • A configuration option to enable non-functional stubs for features that are not implemented on a given architecture. This option for example enables a stub fork() function on non-MMU architectures so that applications can easily be recompiled, without checking all the fork() sites from the beginning
  • Options to enable/disable Linux-specific or BSD-specific functions

The allnoconfig setup with shared library is reported to have been reduced by 30%, though the allnoconfig setup doesn’t necessarily correspond to a classical usage of uClibc.

The tarball is available here.

Crosstool-ng 1.3.0 released!

Crosstool-ng is a tool that allows automated building of cross-compiling toolchain, easing a process known to be very difficult. Crosstool-ng has been started as a rewrite of Crosstool, the famous tool authored by Dan Kegel. Now Crosstool-ng offers several improvements over Crosstool: an active development community, stable releases, support of uClibc, glibc and eglibc, a menuconfig configuration interface, a good documentation, etc.

Yann Morin, the lead developer of Crosstool-ng announced today the release of Crosstool-ng 1.3.0. He says: « There has been many improvements, new features and bug fixes all around. If I had to, my pick would be the support for the gcc 4.3 series. But I would also have to tell you about the latest uClibc version, support for eglibc, and the ability to build bare-metal compilers, and the list would not yet be complete… »

He also mention that SuperH and IA-64 can now build a minimalist C-only toolchain, so the support for these architectures is not complete yet, but progressing. Of course, most components have been updated: new versions, new features, updated patchsets, etc. It for example include support for the latest version of uClibc, 0.9.30, released only two weeks ago.

The Changelog is available, as is a tarball of the new release.

If you need to build some cross-compiling toolchain, you definitely should take a look at Crosstool-ng. It’s great, and well supported: Yann is both very responsive and very helpful when problems are being reported.

Update on flash filesystems

Reviewing new possibilities for flash filesystems – My slides at ELCE 2008

With the release of Linux 2.6.27, including the new UBIFS filesystem for MTD storage, embedded Linux system developers now have multiple choices for their flash storage devices. As far as it is concerned, JFFS2 has also been improved and now has support for LZO compression, which makes uncompressing faster. So, how to choose between JFFS2, YAFFS2, and UBIFS?

To help our customers and the community make the right decision, I measured how these filesystems compare in terms of mount time, access time, read and write speed, as well as CPU usage in several corner cases and with different flash chip sizes.

I showed the results during the Embedded Linux Conference Europe event. Besides sharing lessons learned from these experiments, my presentation also introduced each filesystem and its implementation. I also gave advice for flash based block storage (such as Compact Flash and Solid State disks), to reduce the number of writes and avoid damaging flash blocks.

As usual, Bootlin slides are available under the Creative Commons BY-SA license: flash-filesystems.pdf (PDF), flash-filesystems.odp (Open Document Format).

The main finding is that UBIFS outperforms both JFFS2 and YAFFS2 in almost all corner cases. As shown by the benchmarks, it has consistently good mount time, and read/write performance. If your products are using a recent kernel, and are still based on JFFS2, you should definitely try UBIFS and get significant performance benefits, in particular for boot time, as mounting a JFFS2 root filesystem can take several seconds!

The advent of UBIFS also questions the relevance of YAFFS2. YAFFS2 used to be a good alternative to JFFS2, but unlike UBIFS, it doesn’t support compression. Then, why choose YAFFS2, when a apparently superior alternative is available?

The only case in which JFFS2 can still make sense if when you have very small partitions, sizing just a few megabytes. In this case, the overhead from UBI, the erase-block management layer below UBIFS, is no longer negligible. You will be able to pack much less data than with JFFS2. In this case, you can still improve JFFS2’s performance by using some of its new features (more details in the presentation).

SquashFS is also another great alternative, as shown by my benchmarks. It’s true it is a block filesystem, but since it is read-only, and there is no problem to use it on a write-once mtdblock device. You should really consider it for the read-only parts in your system, though it is advisable to use it on top of UBI, to make its blocks participate to wear-leveling and bad block management. Again, you will find more details in my presentation.

The presentation also mentions LogFS, which is also a promising filesystem for flash storage. Unfortunately, LogFS is not available yet for recent kernels. Stay tuned and I will benchmark it as soon this situation changes.

Embedded Linux From Scratch

An update to this presentation was given in 2019

This presentation shows how easy it can be to build an embedded system from the ground up, rather than trimming an existing general purpose GNU/Linux distribution. It is mainly targeted at beginners in embedded systems, but it also gives useful tricks that more experienced people may not know about.

This document was used in our training sessions. It is available under the Creative Commons BY-SA license (see details and other documents).

It is available under several formats:

Back to our technical presentations

Embedded Linux optimizations

This presentation is a collection of ideas and resources for optimizing the Linux kernel and applications for speed, size, RAM, power and cost. Most of them are gathered and supported by the CE Linux Forum projects. Interested embedded system developers are invited to contribute benchmarks, testing, code and more ideas to these projects.

This document is used in our training sessions. It is available under the Creative Commons BY-SA license (see details and other documents).

It is available under several formats:

Thanks to people who helped, sent corrections or suggestions: Tim Bird, Robert P.J. Day