Buildroot simplified for users!

Buildroot logoYesterday, a set of patches I’ve authored that aims at simplifying Buildroot for users has been merged into the official version of the project, and will therefore be part of the next stable release (scheduled for November, according to our 3 months release cycle). This work is probably my major contribution to Buildroot, outside of external toolchain support and various fixes here and there. Here are quick details about the improvements brought by these patches :

  • Remove the “project” feature. The project feature removal was the main point of this patch set. This feature, that allows to compile a system for different, but very similar platforms, without recompiling everything from scratch, was rarely used and introduced a lot of complexity in the usage of Buildroot for newcomers. Who hasn’t been confused by this project_build_arm directory? This thing is gone now.
  • Remove the BOARD/LOCAL feature, which duplicates another way of adding support for new targets in Buildroot. This is the kind of feature that has been added at the time Buildroot was basically unmaintained, when nobody was able to say « Hey, but you’re just trying to-reimplement something that already exists »
  • Move all output directories in an output directory. By default, when Buildroot is compiled, it generates several directories in the middle of its source code. Now, with this patch, everything is grouped into an output directory, unless out-of-tree compilation is used, of course (with O=)
  • Remove TOPDIR_PREFIX and TOPDIR_SUFFIX since the same effect could already be done using out-of-tree compilation with O=. Another duplicated feature that should never have reached the tree.
  • Rename the output directories. Now that everything is properly stored in an output directory, it was time to rename the subdirectories to make them more meaningful. So now, we have build where all packages are built, images that contains the final binary images of the root filesystem and the kernel, staging which contains the staging directory (all packages installed with their development headers and libraries), target that contains the root filesystem for the target (without the device files), host that contains the installation of the host tools that Buildroot requires for its execution, stamps that contains the stamp files used by Buildroot to keep track of the compilation progress. Therefore, all directories such as build_ARCH or toolchain_build_ARCH have disappeared.
  • Major documentation update, to of course make sure that our documentation is up-to-date with the latest changes.

Getting all these changes mainlined is really a nice thing. I also have tons of other ideas to improve Buildroot infrastructure, and I’m sure the coming Buildroot Developer Day will be a great opportunity to discuss these.

First Buildroot Developer Day after ELCE, on October 17th in Grenoble, France

Buildroot logoThe first Buildroot Developer Day will take place on Saturday, October 17th in Grenoble, France, just the day after Embedded Linux Conference Europe. This Developer Day aims at allowing Buildroot developers to meet and exchange ideas on the project and its future.

As the number of places is limited, interested candidates are invited to send an e-mail to Peter Korsgaard (jacmet at uclibc dot org) and Thomas Petazzoni (thomas dot petazzoni at free-electrons dot com).

Peter Korsgaard and I are organizing this Developer Day thanks to the sponsorship of Calao Systems (offering the location for the meeting) and Free Electrons (offering free lunch to the participants).

See also the announcement of the mailing list and on Buildroot website.

Embedded Linux Conference Europe 2009 program

GrenobleAs we announced a few months ago, Embedded Linux Conference 2009 will take place in Grenoble, France on October 15th and 16th. The list of sessions has just been published online, and as usual, it is a very exciting list of in-depth technical talks.

Because of my involvement in Buildroot, I’m particularly interested in the numerous talks about embedded Linux build systems: Florian Fainelli will talk about OpenWRT (which is not only dedicated to Wifi routers, but is a generic embedded Linux build system, built as a fork of Buildroot), Gordon Hecker on e2factory (a build system I’ve never heard of until now, but conferences are great to discover new tools and projects), Cedric Hombourger on OpenEmbedded (I will be particularly happy to meet Cedric again since I had the chance to work with him six years ago), Marcin Jusziewicz on OpenEmbedded again, Robert Schwebel on PTXdist and finally Alex de Vries on what seems to be a more generic talk about build systems.

Of course, besides build systems, a lot of other topics will be covered. I’ve noted things such as the talk on Canola by Gustavo Barbieri, the boot time presentation by Grégory Clément, the device tree talks by Wolfram Sang and Vitaly Wool, the talk by Alessandro Rubini in order to meet one of the author of Linux Device Drivers, the power management and clock management talks also.

Free Electrons will obviously be present during this conference :

  • Michael Opdenacker, my colleague, will give a talk entitled Update on boot time reduction techniques
  • Michael, again, will be the chair of a Small Business BOF, which should allow small companies offering services around embedded Linux to meet and exchange their ideas and experience
  • Finally, I will be the co-chair with Peter Korsgaard of a BOF on Buildroot
  • We will share a booth with our partner Calao Systems during the Minalogic Embedded Systems Exhibition. This will be another opportunity to meet.

I hope to see you in October at ELCE !

On the road to Buildroot 2009.08

Buildroot logoAs I posted back in March, the Buildroot project has begun a new life since January. One of the aspects this new life is the fact that stable releases are delivered every three months. So we had one in February (2009.02), one in May (2009.05) and therefore we’re going to have one release soon in August (2009.08).

Peter Korsgaard, Buildroot’s maintainer, has just released 2009.08-rc1, a good opportunity to look at what’s new in Buildroot since the last release. To highlight Free Electrons’ participation in the free software community, it is worth noting that in number of patches, I’m the second most important contributor to Buildroot, right after Peter, the maintainer. It’s really nice that I’ve been able to find enough time to contribute at such a level to Buildroot, and hope to be able to dedicate more time in the future to Buildroot, since we have lots of ideas to improve the project.

Back to the 2009.08 release specifically, in the notable features or improvements, we have :

  • Improvement of external toolchain support. I’ve already posted about this specific feature, that I find very important. Several reports from people using this feature have been received since then, which means that it is a useful feature for some of the Buildroot users. So, we have now support for glibc toolchains, the configuration provided to Buildroot is verified against the real toolchain configuration, the copy of the toolchain libraries to the target has been improved (including the C++ standard library) and other minor improvements have been. I’ve been the major contributor of these improvements, together with the reports provided by Buildroot users
  • Integration of the QT-based configurator. Buildroot uses the kconfig system from the kernel to allow its users to configure the different elements of the system to be built. Until now, only the ncurses interface (make menuconfig) was supported. In the 2009.08, the QT interface (make xconfig) will also be supported, adding a little more userfriendliness to Buildroot. This feature was contributed by Alper Yildirim, and I helped in polishing the remaining issues.
  • Support for the Xtensa architecture has been contribued by Maxim Grigoriev, from Tensilica, the company designing the Xtensa architecture.
  • Cleanup of X.org support. the configuration options were clarified, and mandatory dependencies on useless libraries such as libXt or libXaw were removed, some components were upgraded to fix build issues. It’s also a task I contributed to.
  • Of course, toolchain components have been upgraded (GCC 4.4.1, 2.6.30 kernel headers), new packages have been added (bmon, ctorrent, dosfstools, enchant, gst-plugins-bad, iw, libmms, libnl, netstat-nat, ntfsprogs, sdl_gfx, spawn-fcgi), and many more packages have been upgraded (bind, busybox, coreutils, sqlite, directfb, expat, gamin, gnuconfig, haserl, ipsec-tools, classpath, libcurl, libglib2, liblockfile, libpng, libsoup, libxml2, lighttpd, ltp-testsuite, lvm2, matchbox, memstat, gst-plugins-good, gstreamer, libogg, libvorbis, mplayer, neon, openssl, pciutils, php, qt, ruby, sawman, webkit, wpa-supplicant, xdriver_xf86-input-synaptics, xdriver_xf86-video-intel, xlib_libXfont, xlib_libXft, xlib_libXt, xproto_xproto, xserver-xorg, xutil_makedepend, xutil_util-macros)
  • We’re also paying a lot more attention to the bugs reported in our bugtracker than we used to do in the past. Since 2009.05, 43 reported issues have been fixed, and I except that we will fix some more before the 2009.08 release

Shortly after the 2009.08, we will set the goals for the next 2009.11 release, and I’d like to do a lot of cleanup to make Buildroot easier to use and to understand. We’ll see how things go.

In the mean time, if you’re interested in testing and improving Buildroot, don’t hesitate to grab 2009.08-rc1, try it in your conditions with your package set, and report your issues!

Switching between toolchains made easy!

It is quite common to have several toolchains and to switch back and forth between them while doing development. At least, this is something I do a lot when doing Buildroot development and debugging. As I hate typing full paths all the time, I usually put the toolchain bin/ directory into my $PATH variable, so that I can easily access the toolchain binaries. However, it means that everytime you run a new shell or everytime you want to switch from one toolchain to another, you need to modify the PATH variable manually by re-exporting it. Of course, one could easily put all the bin/ directories of all toolchains in the PATH, but that would clutter what is shown when I do arm-TAB-TAB, and that’s not nice.

So, I ended up hacking a few lines of Bash that provide me with two new commands: xtoolsadd, to add a toolchain to my PATH and xtoolsdel, to remove a toolchain from my PATH. These commands work by making the assumption that all toolchains are stored in a common directory. In my case /usr/local/xtools/ contains all the toolchains, one per subdirectory. So I have /usr/local/xtools/arm-unknown-linux-gnu for an ARM glibc-based non-EABI toolchain, or /usr/local/xtools/arm-unknown-linux-uclibcgnueabi for an ARM uClibc-based EABI toolchain).

So, now I can do things such as

xtoolsadd arm-unknown-linux-gnu

or

xtoolsdel arm-unknown-linux-uclibcgnueabi

Because these commands must modify the PATH variable of the current shell, they cannot be implemented as separate shell scripts, so they are in fact implemented as functions in my ~/.bashrc script. And in addition to these functions, I also implemented completion, so when you do xtoolsadd TAB-TAB, it gives you a choice of toolchains, and if you start typing one and press TAB, it will just automatically complete for you. The same thing works with xtoolsdel, of course.

To make this work, here is what you need to put in your ~/.bashrc file:

export XTOOLSDIR=/usr/local/xtools

xtoolsadd() {
    TOOLCHAINDIR=$XTOOLSDIR/$1/bin
    if [ ! -d $TOOLCHAINDIR ] ; then
        echo "Directory $XTOOLSDIR doesn't exist"
    else
        case "$PATH" in
            *"$XTOOLSDIR"*)
                ;;
            *)
                export PATH=$TOOLCHAINDIR:$PATH
        esac
    fi
}

xtoolsdel() {
    TOOLCHAINDIR=$XTOOLSDIR/$1/bin
    NEWPATH=
    found=0
    for i in $(echo $PATH | tr ":" "\n") ; do
        if [ $i == $TOOLCHAINDIR ] ; then
            found=1
        else
            NEWPATH=$NEWPATH:$i
        fi
    done
    if [ $found == 0 ] ; then
        echo "$1 is not in your PATH"
    else
        export PATH=$NEWPATH
    fi
}

_xtoolsadd() {
    cur=${COMP_WORDS[COMP_CWORD]}
    LIST=$(ls -1 $XTOOLSDIR | tr "\n" " ")
    COMPREPLY=( $( compgen -W "$LIST" -- $cur))
}

_xtoolsdel() {
    cur=${COMP_WORDS[COMP_CWORD]}
    LIST=$(echo $PATH | tr ":" "\n" | grep "^$XTOOLSDIR" | sed "s%$XTOOLSDIR/\([^/]*\)/bin%\1%")
    COMPREPLY=( $( compgen -W "$LIST" -- $cur))
}

complete -F _xtoolsadd xtoolsadd
complete -F _xtoolsdel xtoolsdel

The shell code may not be perfect or fully optimized, but it works. Of course, if you have suggestions or questions, don’t hesitate to post comments!

Buildroot gains better support for external toolchains

Buildroot logoBuildroot is a tool that I’ve already covered in a previous blog post. To me, its main purpose is to build the root filesystem for an embedded Linux system, with all the necessary applications and libraries. It automates the tedious process of cross-compiling and integrating all the free software components in an embedded system.

In addition to root filesystem generation, Buildroot is also known for its ability to generate a uClibc-based cross-compilation toolchain. Buildroot used to be for quite some time the only way to generate a toolchain based on this size-effective C library, but it is no longer the case with Crosstool-NG supporting glibc, uClibc and eglibc.

However, I’ve personaly never been really satisfied with uClibc generation of cross-compiling toolchains:

  • It mixes the process of the cross-compilingn toolchain generation with the process of root filesystem generation, which are, in my opinion, two very different processes. Once your toolchain is generated, you generally don’t touch it, but regenerate your root filesystem dozens or hundred of times until all your components are here and properly integrated.
  • The attention paid to toolchain generation in the Buildroot project itself is relatively small, while other projects like Crosstool-NG or vendors like Codesourcery, are specifically dedicated to providing toolchains. The fact that, for example, uClibc is the only C library supported is one example of this.
  • It might necessary, for various reasons, make sense to use an already existing toolchain.

Support for the usage of external toolchains has already been present in Buildroot for a long time, but wasn’t developed enough to be easily usable. Months ago, I’ve started to improve the situation (here, here, here and here), and last week, two other patches have been integrated.

  • The first patch, visible here removes the ugly configuration option that allows to configure the set of libraries that must be copied to the target filesystem, and replaces it with a nice selection of the C libary type: uClibc or glibc. It makes it clear that generating Linux system with the glibc library is possible with Buildroot, even if Buildroot has often been advertised as a uClibc only tool.
  • The second patch, visible here adds checks for the conformity of Buildroot configuration versus the C library configuration. There are configuration options in Buildroot that must tell whether the C library supports IPv6, supports RPC, supports locale, supports large file, etc. These options must be set in the configuration interface according to the C library configuration, because some userspace packages depend on them. The added checks verify that the value set to these options match the configuration of your C library

So, now, external toolchains are a little bit easier to use with Buildroot, and your own vendor toolchain, Codesourcery toolchains or any other toolchain can be used with Buildroot. The only requirement is that the toolchain supports the sysroot feature, which is very common in most toolchains.

Linux kernel 2.6.29 – New features for embedded users

Tuz Linux logoThe 2.6.29 version of the Linux kernel has just been released by Linus Torvalds. Like all kernel releases, this new version offers a number of interesting new features.

For embedded users, the most important new feature is certainly the inclusion of Squashfs, a read-only compressed filesystem. This filesystem is very well-suited to store the immutable parts of an embedded system (applications, libraries, static data, etc.), and replaces the old cramfs filesystem which had some strong limitations (file size, filesystem size and limited compression).

Squashfs is a block filesystem, but since it is read-only, you can also use it on flash partitions, through the mtdblock driver. It’s fine as you just write the filesystem image once. Don’t hesitate to try it to get the best performance out of your flash partitions. Ideally, you should even use it on top of UBI, which would transparently allow the read-only parts of your filesystem to participate to wear-leveling. See sections about filesystems in our embedded Linux training materials for details.

This new release also adds Fastboot support, at least for scsi probes and libata port scan. This is a step forward to reducing boot time, which is often critical in embedded systems.

2.6.29 also allows stripping of generated symbols under CONFIG_KALLSYMS_ALL, saving up to 200 KB for kernels built with this feature. This is nice for embedded systems with very little RAM, which still need this feature during development.

Another new feature of this kernel is the support for the Samsung S3C64XX CPUs. Of course, a lot of new boards and devices are supported, such as the framebuffer of the i.MX 31 CPU, or the SMSC LAN911x and LAN912x Ethernet controllers.

The other major features of this new kernel, not necessarily interesting for embedded systems are the inclusion of the btrfs filesystem, the inclusion of the kernel modesetting code, support for WiMAX, scalability improvements (support of 4096 CPUs!), filesystem freezing capability, filesystem improvements and many new drivers.

For more details on the 2.6.29 improvements, the best resource is certainly the human-readable changelog written by the KernelNewbies.org community.

The Buildroot project begins a new life

The Buildroot project has been around for a quite a while. For those just discovering Buildroot, it is a set of Makefiles that ease and automate the process of building a cross-compilation toolchain (based on the uClibc C library, since Buildroot has been initiated by uClibc developers) and a full embedded Linux system. Buildroot can compile over 600 packages : graphical libraries (Qt Embedded, Gtk, X.org, DirectFB), network servers (the Dropbear SSH server, several HTTP servers), and more), core components such as Busybox, DBUS, Avahi and many other free software packages that make sense on embedded systems. Of course, more packages can easily be added. Buildroot is very similar to PTXdist.

I remember using to this project and contributing to it back in 2004 when I still was an intern during my studies. One of my contributions was the writing of documentation for the project, which is still the official documentation. Since 2004, the project has evolved, but there was no clear maintainer and no stable releases. The developer community around Buildroot was not completely satisfied since no one was merging the proposed fixes and improvements, since there was no official maintainer. And Buildroot was difficult to use for users because it didn’t offer any kind of stable releas : users had to pick a random SVN checkout and cross fingers to get a reasonably-working version.

In January 2009, as part of the traditionnal new year’s resolution, Peter Korsgaard, one of the most active Buildroot developers, volunteered to step up as the official maintainer and to deliver releases. And indeed, he integrated many patches that were floating around, released several release candidates before releasing Buildroot 2009.02 on February, 12th. Since then he continues to make very interesting improvements to Buildroot, replies on the mailing list, review and merge the proposed patches. A new bug tracker has been set up, and the source code will soon be moved over the Git distributed version control system.

Definitely, it’s a new life starting for the Buildroot project. As a long-time user and occasional contributor of the project, I’m glad to see such evolution.

FOSDEM 2009 videos

As previously announced, I had the chance to attend the FOSDEM conference again this year. And once again, the famous Free Electrons video camera was with me, and I could record a few talks. I’m pleased to make them available today.

FOSDEM, the Free and Open Source Software Developers' European Meeting

Here are the available videos, with my personal comments. Of course, except the talk about CMake, all of them come from the embedded devroom. Thanks to the speakers for sharing their knowledge and presenting their projects !

  • video captureWt, a C++ web toolkit, for rich web interfaces to embedded systems, by Pieter Libin (Noesis Solutions)
    Video (44 minutes, 124M)
    Presentation of a Web application generator, which generates a Web application including a web server from a C++ description. The C++ design is based on many ideas coming from Qt, which eases the creation of the graphical application. Certainly a very interesting tool for embedded systems.
  • Hacking with modular hardware: the BUG, by Ken Gilmer (Bug Labs)
    Video (43 minutes, 129M)
    The BUG is a set of hardware modules that can be combined together. A base module contains the CPU, memory and other basic components, while additional modules can provide GPS, Wifi, webcam, I/O lines and many others. Of course, it comes with a completely free software SDK. Can be an interesting starting point for prototyping or hobbyist embedded hacking, even though the hardware is quite expensive.
  • Building Embedded Linux Systems with PTXdist, by Robert Schwebel (Pengutronix)
    Video (68 minutes, 151M)
    A very good presentation on why system building tools are needed for Embedded Linux systems (cross-compiling issues, etc.), and how PTXdist fits this need. Even though I’m personally quite fan of Buildroot, PTXdist’s competitor, the presentation was very interesting.
  • video captureDevelopment on the OpenMoko with hackable:1, by Pierre Pronchery (Bearstech)
    Video (55 minutes, 189M)
    A hands-on lab on the OpenMoko, which unfortunately was too short to be really interesting, and restricted to only the 10-12 people at the front of the room. Interesting for the people who did it, probably not so interesting to watch afterwards.
  • Development and Certification of Linux-Based Fire Safety & Security Systems, by Baurzhan Ismagulov (Siemens)
    Video (47 minutes, 124M)
    The topic of the talk was really appealing, because the use of free software in environments where human life is in danger has always caused many debates and discussions. Unfortunately, the talk completely misses the point : the speaker spent the whole talk discussing Germany-specific rules and laws for the certification of fire safety and security systems, almost without saying a word on how Linux can match these rules.
  • Maemo on BeagleBoard, by Juha Kallionen (Nokia)
    Video (20 minutes, 51M)
    Good talk, but not a lot of content, outside of « Hey, we run Maemo on BeagleBoard, try it !». A kind of announcement talk, I would say.
  • video captureAdvanced power management for OMAP3, by Peter De Schrijver (Nokia)
    Video (49 minutes, 169M)
    A very interesting, in-depth, technical talk about the power management features of the OMAP3 CPU and how these features can be used by the Linux kernel.
  • Emdebian 1.0 release – small and super small Debian, by Neil Williams (Debian)
    Video (86 minutes, 257M)
    video captureAgain, another interesting talk about the status of Emdebian, which has just reached 1.0. Neil described both Emdebian Crush and Emdebian Grip, that are two different approaches for making Debian more suitable for embedded systems. Very good progress has been made, and Emdebian is certainly something that should now be considered as a distribution for embedded systems.
  • CMake – what can it do for your project, by Alexander Neundorf
    Video (45 minutes, 167M)
    A general introduction to CMake, with lots of examples and demonstrations. Very nice for those who never had the opportunity to look at CMake.

Program for Embedded Linux Conference 2009 announced

CELF penguinThe program for the Embedded Linux Conference 2009 has been announced a few days ago, and the registration is now open. For the record, the Embedded Linux Conference is probably the largest technical conference specifically dedicated to the use of Linux on embedded systems. Organized by the CE Linux Forum every year, this conference gathers a large audience and a wide range of very interesting talks.

The 2009 edition of ELC will take place from April 6th to April 8th, in downtown San Francisco, in the United States.

The program features :

  • Almost 50 talks, tutorials and keynotes. The topics go from multimedia, flash filesystems, system initialization, memory management, instrumentation and debugging tools, real-time, embedded distributions, system building,
  • An opening keynote by Dirk Hohndel from Intel
  • A keynote by David Woodhouse, one of the two embedded Linux maintainers
  • Other famous speakers such as Paul Mundt (maintainer of the Linux port to the SuperH architecture), Kate Alhola (from Maemo/Nokia), Dan Malek (from Embedded Alley), Mike Anderson (the PTR group, famous for his JTAG tutorial), Jake Edge (from LWN), Klaas van Gend (from Montavista), Frank Rowand (from Sony, famous for his real-time adventures talk), Jim Ready, Denis Oliver Kropp (DirectFB main developer)
  • A panel on The Linux Kernel, what’s next with Jonathan Corbet, Greg KH, Andrew Morton, Keith Packard and Ted Ts’o
  • And last but not least, a talk from Michael Opdenacker, Free Electrons, on flash filesystems.

Of course, both Michael and I will attend the conference. Hope to meet you there!
Golden Gate Bridge