ELC Europe 2010 sessions announced

List of sessions and speakers at ELC Europe in Cambridge, UK

Cambridge, UKBeing a member of the organization committee of the Embedded Linux Conference Europe, I get access to fresh news about this yearly conference. The call for presentations is now over and we have just announced the list of sessions.

Note that this list is not the final one yet. Some speakers haven’t confirmed their participation or haven’t sent their biographies yet. There are also two or three speakers added at the last minute who are not listed yet.

The conference will happen in Cambridge, UK, on October 27-28, 2010. Keep an eye on the website (or on our blog). Registration should open in a few days from now, and all practical details will be given then.

See also the agenda of the GStreamer conference which will happen at the same location on the day before.

How to find the root device?

How to find which device corresponds to your root filesystem

I recently found something I was looking for for quite a long time. If you use the mount command in Linux, you can see that the root device is not listed like the other mounted filesystems:

/dev/root on / type ext3 (rw)
/dev/mmcblk0p1 on /mmcboot type vfat (rw)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /dev type tmpfs (rw,mode=0755)
...

For the / mount point, you are just told that it corresponds to /dev/root, which is not the real device you are looking for.

Of course, you can look at the kernel command line and see on which initial root filesystem Linux was instructed to boot (root parameter):

$ cat /proc/cmdline
mem=512M console=ttyS2,115200n8 root=/dev/mmcblk0p2 rw rootwait

However, this doesn’t mean that what you see is the current root device. Many Linux systems boot on intermediate root filesystems (like initramdisks and initramfs), which are just used to access the final one.

I explored the contents of /proc, but didn’t find any file revealing what the root device is.

Fortunately, I eventually found a command to find the root device:

$ rdev
/dev/mmcblk0p2 /

But how does this work? How could we find such information by ourselves? Use the Source, Luke!

When you ask yourself questions like this one, the best is to look at the BusyBox sources which implement this command. These sources are usually simpler than the ones for the same GNU command.

Here is what BusyBox rdev does… It first runs the stat system call on the / directory. Let’s run the stat command that corresponds to it:

$ stat /
  File: `/'
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d	Inode: 2           Links: 23
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-07-21 22:00:01.000000000 +0200
Modify: 2010-06-13 15:04:37.000000000 +0200
Change: 2010-06-13 15:04:37.000000000 +0200

What’s interesting is the Device field. It means that the device corresponding to / has the major number b3 in hexadecimal (179 in decimal), and minor number 02. Bingo, this corresponds to /dev/mmcblk0p2:

$ ls -l /dev/mmcblk0p2 
brw-rw---- 1 root disk 179, 2 Jan  1  1970 /dev/mmcblk0p2

Therefore, what BusyBox rdev does is walk through /dev and its subdirectories to find a device file matching the major and minor numbers.

This is not a completely generic solution though. On some very simple embedded systems, you don’t even need to create device files for all existing devices. In particular, the device file for the root filesystem doesn’t have to exist. In such a case, rdev wouldn’t be able to find the root device.

A more generic solution could be to walk through /sys/block which enumerates all the block devices present on a system (even if not all of them have an entry in /dev/. This would allow to find the device with the matching major and minor numbers:

$ cat /sys/block/mmcblk0/mmcblk0p1/dev
179:1

Through this example, you can see how useful it can be to study the sources of system commands to understand how the system works. BusyBox sources, implementing simplified versions of GNU utilities, make this even easier.

Ubuntu 10.04 on the IGEPv2 board

Installing Ubuntu 10.04 on the IGEPv2 board. Using the board as a small server.

IGEPv2 boardLast year, folks at Texas Instruments told me about the IGEP v2 board. This board is similar to the Beagle board, but also features 512 MB of RAM and NAND flash (instead of 256 for the Beagle), on board Ethernet (RJ45), Wi-Fi and Bluetooth, all this for only 145 €! Its fast ARM CPU (TI OMAP 3530 running at 720 MHz) and graphical capabilities allow it to be used in for services usually performed by desktop or server CPUs.

At the Bootlin main office, we needed a server to share files, create backups and upload these backups to our servers on the Internet. I decided to use the Ubuntu 10.04 distribution on ARM, based on Debian GNU/Linux. As I didn’t find all the details I needed on the IGEP community website, here are the steps that I took. Several details were found on the http://labs.igep.es/index.php/How_to_get_the_Ubuntu_distribution page though.

This page assumes that you are familiar with building the Linux kernel, controlling an embedded board from a serial line and booting it, and using the GNU/Linux system in general (see the training materials from our embedded Linux course). Beginners may be lost because we don’t give all the details, but more experienced developers should just find the board specific details that they need.

First, get an SD card (at least 2 GB), and prepare its partitions with the mkcard.sh utility.

To compile your kernel, get a CodeSourcery toolchain for ARM. I used the 2010q1 release. Install it in /usr/local/CodeSourcery/arm-2010q1/ (for example)

Get the kernel sources:

$ mkdir $HOME/igep
$ cd $HOME/igep
$ git clone git://git.igep.es/pub/scm/linux-omap-2.6.git
$ cd linux-omap-2.6/

Let’s switch to the latest stable version:

$ git tag
v2.6.28.10-3
v2.6.28.10-igep0020b-0
v2.6.28.10-igep0020b-1
v2.6.28.10-igep0020b-2
v2.6.33.2-0
v2.6.33.4-0
$ git checkout -b v2.6.33.4-0 v2.6.33.4-0
Checking out files: 100% (13116/13116), done.
Switched to a new branch 'v2.6.33.4-0'

Now, set the environment variables for cross-compiling the kernel sources to the arm architecture:

$ export PATH=/usr/local/CodeSourcery/arm-2010q1/bin:$PATH
$ export ARCH=arm
$ export CROSS_COMPILE=arm-none-linux-gnueabi-

Now, take the default configuration for the board and build your kernel:

$ make help | grep igep
$ make igep0020_defconfig
$ make -j 4
$ make uImage

It’s time to build your Ubuntu filesystem, using the Rootstock utility:

$ tar zxvf rootstock-0.1.99.3.tar.gz
$ cd rootstock-0.1.99.3
$ sudo ./rootstock --fqdn igepv2 --login mike --password letmein \
  --imagesize 2G --seed build-essential,openssh-server --dist lucid

Copy the kernel to the first partition of your SD card:

cp arch/arm/boot/uImage /media/boot/
cp .config /media/boot/config-2.6.33.4

Install the root filesystem on the second partition of your SD card:

$ cd /media/rootfs/
$ sudo tar zxvf $HOME/igep/rootstock-0.1.99.3/armel-rootfs-201006102239.tgz

Configure the rootfs to let you log in on the serial console (ttyS2 with OMAP). Do this by copying etc/init/tty1.conf to etc/init/ttyS2.conf and replacing tty1 by ttyS2 in this file.

Install kernel modules manually for the first time:

$ mkdir -p /lib/modules
$ cd $HOME/igep/linux-omap-2.6/
$ make INSTALL_MOD_PATH=/media/rootfs modules_install

In the Rootstock version I tested, the specified user didn’t get created (bug report). To be able to log in, I had to disable the root password by removing the first * character in the root entry in etc/shadow:

We are now ready to boot our new system. First, unmount your SD card partitions:

$ sudo umount /media/boot
$ sudo umount /media/rootfs

Insert your SD card in the slot on your board, connect your serial cable and in the U-boot prompt on the serial line, configure the kernel boot parameters:

$ setenv bootargs mem=512M console=ttyS2,115200n8 root=/dev/mmcblk0p2 rw rootwait
$ setenv bootcmd 'mmc init 0 ; fatload mmc 0 80000000 uImage ; bootm 80000000'
$ setenv autostart yes
$ saveenv
$boot

You should see your Linux kernel boot and get to a login shell. Log in as root with no password.

It is now time for the final tweaks. First, create a non root user (remember the Rootstock bug), allow it to run the sudo command, and choose a root password too:

adduser mike
adduser mike sudo
passwd

Let’s cope with a last Rootstock bug. Add the updates and security repositories to /etc/apt/sources.list:

deb http://ports.ubuntu.com/ubuntu-ports lucid-updates main
deb http://ports.ubuntu.com/ubuntu-ports lucid-security main

Without this, you would miss package updates and security releases, and your packages would never change!

If you use the IGEP board as a server as I do, you may need your server to have a fixed MAC address. The trouble is the e2prom storing the MAC address is not populated by default, and every time you boot, the kernel gives you a random MAC address.

The easiest fix I found was to choose an arbitrary MAC address (you can take the first random one that you get), and force it in /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp
hwaddress ether 00:01:04:1b:2C:1F

As the IGEPv2 board doesn’t have a battery by default, it won’t be able to keep the correct time. You can use the ntp daemon to address this:

sudo apt-get install ntp

Your configuration should now be complete. You can now use your IGEPv2 board as a tiny, ultra low power server with Ubuntu server. All the rest is ordinary Debian / Ubuntu server administration. Of course, you can also install desktop packages and use your board as a desktop replacement (you may need to add kernel command line settings for graphics). Have fun!

By the way, the IGEPv2 board is not the best solution if you all you need is a server. The amazing graphical capabilities of the OMAP chip would just be useless. For a server, better, cheaper and more powerful alternatives are the SheevaPlug and GuruPlug. Don’t miss these very nice devices!

Linux device drivers architecture talk at Libre Software Meeting

recursive device modelThomas Petazzoni gave a talk on the Linux kernel architecture for device drivers at the Libre Software Meeting in Bordeaux, France. While the talk was given in French, the materials are in English and can therefore benefit a larger audience. The talk seems to have been well-received, especially from people already having a basic Linux kernel development experience. The topics covered are part of our Linux Kernel development training, and are also usually very appreciated from the trainees already having Linux kernel experience.

The idea of the talk is to give an overview of how device drivers fit into the kernel, both to expose their functionality to upper layers (such as a network device driver exposes itself to the kernel network infrastructure) and to detect/access the hardware using the device/driver model, which is quite hard to understand from the source code only.

The talk went through the following sections :

  • First a basic introduction to device drivers: how devices are seen from userspace applications, and how a simple, raw, character driver can be implemented. It allowed to expose the principle of operations and their similarity with methods in object-oriented programming, and the principle of registration to an upper-layer infrastructure
  • Then, an introduction to what I call « kernel frameworks », i.e kernel subsystems that specialize a general device type (i.e character device) into a particular device type (i.e serial port device, framebuffer device, etc.). The talk illustrates this with the framebuffer core and the serial port core.
  • Finally, an explanation about the device model: bus drivers, adapter drivers and device drivers. I started with the example of the USB bus: being a dynamically-enumerated bus, it provides a good illustration of the device model principles. At the end, I explained how the device model works for the devices embedded into a SoC using the platform drivers/devices mechanism

The slides for this talk are no longer available, but their updates are now integrated in our Linux kernel and driver development training materials which are freely available.

Embedded Linux Opportunities – Keynote in Eindhoven

Fontys UniversityIf you live near Eindhoven in the Netherlands, or if you can easily get there, you may be interested in the seminar organized by Fontys University on June 25, 2010.

I will be in Eindhoven on that week to give our Embedded Linux System Development course. The session is organized by Fontys University and Alten PTS, and Fontys invited me to give a keynote at their seminar on Friday.

The seminar topic is: “Embedded Linux for SMEs” – A guide to professional development:

  • Ruud Ermers (Fontys) – A Guide to professional development
  • Gerben Blom (Alten PTS) – Android for SMEs
  • Michael Opdenacker (Bootlin) – Embedded Linux Opportunities. Ideas and advise for the Open Minded.
  • The grand race, between arm9 controlled vehicles.
  • Technical demos

Here is the abstract for my speech:

Michael Opdenacker created Bootlin in 2004, a company that is best known worldwide for the technical resources it shares with the embedded Linux community. The more this company shares (in the true sense of sharing: giving and receiving), the more successful it gets.

That’s why Michael will come to exchange ideas and experience on building successful embeddded Linux systems and businesses. Just make sure you send him a SIGSTOP signal at the end of his timeslot.

To water your mouth, this talk will show a few things which are possible with embedded Linux and free software building blocks. You will see how you can turn an idea into something that could keep you busy during your whole life if you are not careful.

So, if you are in the Eindhoven area, this could be a nice opportunity to meet. I hope to see you there!

Buildroot 2010.05 released: again many Bootlin contributions

Buildroot logoJust one day before the end of May, Buildroot 2010.05 has been released by Peter Korsgaard, as predicted by the fixed release schedule used by the project. It can be downloaded at http://buildroot.org/downloads/buildroot-2010.05.tar.bz2. For the record, Buildroot is a simple and efficient tool to build embedded Linux systems: cross-compiling toolchain, root filesystem, kernel image and bootloader.

Major changes

The major user-visible changes are:

  • Re-organization of the menuconfig layout for packages. All packages are now organized in categories, making them easier to find
  • Our X.org package set has been upgraded to X.org 7.5.
  • Several new packages have been added: cdrkit, cramfs, genext2fs, genromfs, libatomic_ops, librsync, libusb-compat, lmbench, netperf, squashfs, squashfs3, squid. Many of them have been added as the result of a filesystem code generation cleanup
  • On the internal toolchain side (i.e toolchains generated by Buildroot), we have added support for uClibc 0.9.31, GCC 4.4.4, GDB 7.x and binutils 2.20.1.
  • On the external toolchain side (i.e. re-using existing toolchains), we have improved support for multilib toolchains (such as CodeSourcery toolchains)

In addition to these changes, 41 bugs of our bug tracker have been fixed, and dozens of packages have been upgraded or fixed.

Bootlin contributions

Bootlin has again made significant contributions to this release:

git shortlog -s -n 2010.02..
   224  Paulius Zaleckas
   182  Thomas Petazzoni, from Bootlin
   148  Peter Korsgaard
    28  Gustavo Zacarias
    26  Will Wagner
    14  Lionel Landwerlin
     6  Yann E. MORIN
[...]

The things we have contributed include:

  • A big cleanup in the Buildroot code that generates the root filesystem images. It has been moved from various directories in target/ to a single, central location: fs/. The code that handled the compilation of host utilities to generate the filesystem images (genext2fs, cdrkit, mtd-utils, cramfs, squashfs, etc.) has been moved to normal packages, and an infrastructure has been added to factorize the common code of the various filesystem generation makefiles.
  • Better support for multilib external toolchains
  • A new script that generates nice dependency graph (see below)
  • A clarification of the gettext integration, to make it work properly with glibc toolchains.
  • Fixes to bug #75, bug #1789.
  • Dozens of build fixes found by testing random configurations.
  • Various code cleanups, that lead to the removal of several configuration options, which makes the usage of Buildroot a little bit easier.

Dependency graph generation

Thanks to the new package infrastructure that we have included in Buildroot a few releases ago, it is now easier to retrieve the list of dependencies of each package in a generic way. Using this, I recently implemented a dependency graph generation tool. It allows to generate nice graphs of the dependencies for a given package, like libgtk2 in the following example (click for the full sized version):

Note that packages in red are packages that do not use the generic or autotools infrastructure, so we couldn’t determine what their dependencies are.

We can also generate the dependency graph for a complete Buildroot configuration, with all packages:

Using this tool is fairly easy. You must first install the graphviz package on your distribution. For a single package dependency graph:

./scripts/graph-depends libgtk2 > libgtk2.dot
dot -Tpdf libgtk2.dot -o libgtk2.pdf

For a full dependency graph:

./scripts/graph-depends > full.dot
dot -Tpdf full.dot -o full.pdf

Note that the dependency graph always depends on the selected set of packages. It is not the absolute dependency graph, which would contain all existing dependencies. It only show the dependencies as they are in your current configuration.

Plans for the next release

The next release is scheduled for August (2010.08). I have in mind various things:

  • Cleanup of the bootloader compilation code and integration of support for Barebox. This is already implemented in one of my branch, so I should be able to push this fairly soon to Buildroot
  • Cleanup of the Linux kernel compilation code, with a much-wanted simplification of this. Again, this is already implemented on my side, but requires a little bit more work before being able to push this
  • Continue the effort to convert packages to the generic or autotools infrastructure. I have already sent a status update on this topic to the project mailing-list. We have 56 packages to convert to the generic infrastructure and 77 to convert to the autotools infrastructure.
  • Integration with Crosstool-NG, which is a job that Yann Morin, developer of Crosstool-NG has already started.
  • More improvements of the external toolchain integration
  • If some time is left, a cleanup and reorganization of the board support mechanism, so that we can add more boards in a sane way

Issues with changing Ethernet device names

Beagle boardNo eth0 interface in your system!

If you are using Ubuntu or Debian on ARM, as I am doing at the moment, you may be surprised to see that your system has an eth1 interface (or even eth5), but no eth0!.

This also happens on x86, of course, and probably with other distributions. I faced and understood this problem when I migrated a KVM virtual machine from a regular network interface card to a virtio one. eth0 was gone, and because my network settings were bound to this interface, I couldn’t connect to my virtual machine anymore.

In recent releases of Ubuntu, this happens because of some udev rules, which try to make sure that each network card always gets bound to the same ethx device. So, every time the system see a new network card, the /etc/udev/rules.d/70-persistent-net.rules file gets updated as follows:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x14e4:0x167a (tg3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:18:8b:87:52:12", ATTR{type}=="1",
KERNEL=="eth*", NAME="eth0"

# USB device 0x:0x (asix)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:00:eb:eb:26", ATTR{type}=="1", KERNEL=="eth*",  NAME="eth1"

This can happen if you replace a USB to Ethernet dongle by another one with a different MAC address, or if you reuse a filesystem built on another board. Even if the board and Ethernet device are the same, you will get eth1 instead of eth0. This can be annoying if you configured your network through /etc/network/interfaces or if you bring up your network with your own scripts.

A clean solution for this problem is simply to remove the /etc/udev/rules.d/70-persistent-net.rules file. The next time you boot, you will get back to a normal eth0 network device.

I checked that these persistent net naming rules are not part of udev releases. In other distributions, you could have udev without such rules.

Location of Embedded Linux Conference Europe 2010 announced

Cambridge, UKWhile the American Embedded Linux Conference always takes place in the San Francisco area in California, the European Embedded Linux Conference Europe travels each year in a different country across Europe: it took place in Austria in 2007, in the Netherlands in 2008 and in France in 2009. The location for the 2010 edition of ELCE has been announced recently: it will take place on October 27th and 28th in Cambridge, United Kingdom. As usual there will be many talks, Bird-of-a-Feather sessions, technical demonstrations and more. If you’re an embedded Linux developer in Europe, you should definitely ask your employer to send you to this conference!

Presentations on the following topics are encouraged: audio, video, and graphics systems for embedded products, security, system size, boot-up time, meeting real-time constraints, power management, streaming media, flash memory devices and filesystems, technologies related to cell phones, digital set top boxes, handheld devices, or other CE products, development tools for embedded users, use of Linux in actual products, practical experience and war stories, standards for CE products. The proposal must be received by June, 30th. My colleague Michael Opdenacker, founder of Bootlin, is a member of the program committee.

On the day before, note that there will also be an interesting event: GStreamer Conference 2010. As GStreamer is a key component of today’s multimedia systems based on embedded Linux. Staying in Cambridge for three days instead of two could then make a lot of sense.

Crosstool-NG 1.7.0, Bootlin contributed basic Blackfin support

Yann E. Morin, maintainer of Crosstool-NG, has just announced version 1.7.0 of his toolchain building tool we like so much at Bootlin. For the record, Crosstool-NG allows to build cross-compiling toolchains for a wide-range of architectures, using either glibc, uClibc, eglibc or newlib, and everything is configured through a menuconfig interface, familiar to all embedded Linux developers.

Blackfin architectureBootlin has contributed to this release, by adding some support for the Blackfin architecture, on which we have been working recently. Crosstool-NG is now capable of generating a FDPIC cross-compiling toolchain for Blackfin. FDPIC allows to use shared libraries even on no-MMU architectures like the Blackfin. This support of Blackfin in Crosstool-NG is only the beginning, a lot of work remains to be done to make sure that the toolchain works really well, and to support FLAT binaries and bare-metal as well.

Here are the patches that Bootlin has contributed for this release :

The other major changes for this release are :

  • two new architectures have been added: Blackfin and mips64
  • building canadian-crosses has been enhanced
  • experimental support for gcc-4.5 has been added
  • eglibc can be optimized for size (instead of speed)
  • companion libraries can be built statically, removing the need for a wrapper
  • many components versions have been added, and associated patchsets updated
  • some less important new features, or enhancements for stability and usability
  • a lot of bug-fixes, of which some to help on Darwin & *BSD

The 1.7.0 release can be downloaded at http://ymorin.is-a-geek.org/download/crosstool-ng/crosstool-ng-1.7.0.tar.bz2 and the homepage of the Crosstool-NG project is http://ymorin.is-a-geek.org/projects/crosstool. Enjoy !

New jobs at Bootlin

Penguin worksLooking for kernel and embedded Linux experts

Bootlin is looking for experienced members of the Free Software community to satisfy increasing demand for development, consulting and training on embedded Linux and on the Linux kernel.

One thing that distinguishes our positions from others is that contributing to the community will be part of your objectives.

All the details can be found on our careers page.