Linux Kernel Development, third edition

Linux Kernel Development, by Robert Love, 3rd edition

Linux Kernel Development is a book authored by Robert Love, a famous kernel developer. Contrary to the very famous Linux Device Drivers book, Linux Kernel Development is not oriented towards driver development, but instead covers how the core Linux kernel works.

Having this knowledge is not absolutely necessary to write Linux device drivers, but having a good overall understanding of the kernel always help to understand what’s going on in your system, even at the application level. In July this year, the third edition of Linux Kernel Development has been published, which upgrades the book contents to kernel version 2.6.34, a good opportunity to have a new look at the book that Bootlin received a few weeks ago.

After a quick introduction to the kernel sources (configuring, building, organization of the source tree), the book immediately dives into kernel internals:

  • Process management: how the kernel represents processes and their state, how processes are created inside the kernel, how threads are handled, are processes are terminated.
  • Process scheduling: a full chapter dedicated to the Linux kernel process scheduler. The new CFS scheduler is of course covered in great detail, with large portion of commented source code, for those who want to understand the fine details of the scheduler. Topics such as process sleep/wake-up, preemption, real-time scheduling policies are also covered.
  • System calls are then covered: how they are implemented, how parameters are passed from userspace to the kernel, etc. The call path from your user-space application down to the kernel is well explained in this chapter.
  • Kernel data structures: a generic chapter which details the kernel API for linked lists, queues, maps, and binary trees. Those APIs are omni-present inside the kernel, and it’s therefore a good idea to know how they work, both for understand existing code and for writing new code.
  • Interrupts: how interrupts are handled and how one can write an interrupt handler. Unfortunately, the newly introduced threaded interrupt handlers are not covered, but it’s true that their usage is not yet very widespread inside the mainline kernel.
  • Bottom halves and deferring work, a topic closely related to interrupt handling. It covers bottom halves, softirqs, tasklets and workqueues.
  • Kernel synchronization: two chapters are dedicated to this topic. First a chapter detailing why synchronization is needed, what are the sources of concurrency and what should be protected against concurrent access. And then a chapter detailing the mechanisms provided by the kernel to implement proper synchronization: atomic operations, spin locks, reader-writer spin locks, semaphores, reader-writer semaphores, mutexes, completion variables, sequential locks, preemption disabling, ordering and barriers
  • Timers and time management details how the kernel manages time: ticks, jiffies counter, timers, delaying execution of code are covered in this chapter. There are unfortunately no details about the clocksource and clockevents infrastructure, and no details about how timers and high-resolution timers are implemented. Contrary to other chapters that go fairly deep into the implementation details, this one mostly only covers the API to time management rather than the internals.
  • Memory management is the topic of the following chapter: physical memory management with the page allocator and the physical zones, then the kmalloc, vmalloc and SLAB allocators are covered. High-memory mappings, a topic specific to 32 bits architectures having more than a gigabyte of RAM is also covered in detail. The per-cpu interface is also covered, and will help those who want to understand parts of the kernel that have been optimized for scalability on multiple CPUs.
  • The Virtual Filesystem, with its different objects: superblock, inode, dentry and file is covered in good detail.
  • The block layer is then covered, with a description of the role of the bio structure, the request queues, and the I/O schedulers.
  • Then, the book goes back to more details about the internals of memory management: the mm_struct memory descriptor, the virtual memory areas (so called VMAs) and how they relate with the mmap/munmap system calls. The next chapter continues with a detailed description of the page cache implementation.
  • A fairly strange chapter called “Devices and modules” gives some information about kernel modules (how to build them, how to use module parameters, how dependencies are handled), then covers the internal of the device model (kobjects, ktypes, ksets, krefs) and finally sysfs. Just like the chapter covering the device model in the third edition of Linux Device Drivers, I think it totally misses the point. All the kobject, ktypes, ksets and krefs stuff is very low-level plumbing used by the Linux device model, but it is not exactly what the driver developer needs to interact with in the first place. In my opinion, a good description of the device model should rather explain what struct bus_type, struct driver and struct device are, how they are specialized for the different bus types in pci_driver, pci_device, usb_driver, usb_device, platform_driver, platform_device, and how the registration/probing of drivers and devices is done. I’ve recently given a talk about this topic, the video is in French, but the slides are in English.
  • Debugging is then covered: printk of course, but also oopses, debugging-related kernel options, the magic SysRq key, kernel debuggers, etc.
  • A rather generic chapter about Portability is then proposed, and finally a chapter about Patches, Hacking and the community that details the kernel community, the kernel coding style, how to generate and submit patches, etc.

All in all, Linux Kernel Development remains very good reading. I particularly appreciate the writing style of Robert Love, who manages to make a deeply-technical book interesting and easy to read. Of course, there are some topics in the kernel in which I had to dive myself and for which I’d expect to see more details in this book, but giving every possible detail about a huge beast such as the Linux kernel in just 400 pages is not possible!

Videos from FOSDEM 2010

Peter Korsgaard presenting Buildroot in the Cross build systems workshop at FOSDEM 2010

Like every year, the Free and Open Source Developer European Meeting took place early February in Brussels, and Thomas Petazzoni, from Bootlin, attended and recorded a few talks from the embedded session. However, contrary to previous years, I haven’t been able to record all talks from the embedded session, since I attended talks from other sessions which were already being recorded by others.

Gian-Carlo Pascutto presenting Embedded software development best practices at FOSDEM 2010

We also attended talks from the and Coreboot developer roooms : videos for the developer room can be found at and videos for the Coreboot developer room can be found at

ELC 2010 videos

Videos from the Embedded Linux Conference in San Francisco, April 12-14, 2010.

The 2010 edition of the Embedded Linux Conference was once again a very interesting event. For embedded Linux developers, the Embedded Linux Conference is a perfect place to learn about new technologies, profit from the experience of other developers, and to meet key software developers.

For people who couldn’t attend this conference, and for single core people who didn’t manage to attend two or three talks at the same time, here are the videos that we managed to shoot. As usual, the videos are released with a Creative Commons Attribution – ShareAlike 3.0 license.

We hope it makes you feel like joining the next edition of the conference. If you can’t wait, what about going to ELC Europe in Cambridge (UK) in late October? It has a very interesting program too. Of course, the sessions will also be recorded. I hope to see you there!

Buildroot 2010.08 released!

Buildroot logoOn the last day of August, just in time, the 2010.08 version of Buildroot has been released. For the record, Buildroot is an easy-to-use embedded Linux build system: it can build your toolchain, your root filesystem with all its components (Busybox, libraries, applications, etc.), your kernel and your bootloaders, or any combination of these components.

Amongst the interesting changes in this version :

  • Complete rewrite of the bootloader build code. It contained a lot of legacy, unused and unclear stuff, it is now much easier to use and extend. We’ve removed support for Yaboot and added support for the new Barebox bootloader, and all the code to support AT91Bootstrap, AT91DataFlashBoot, U-Boot, Grub and Grub 2 has been rewritten.
  • Complete rewrite of the Linux kernel build code. It was also complicated to use, with an horribly complicated kernel version selection mechanism, the new code is much easier to configure and use.
  • The configuration file .config is now located in the out-of-tree directory when the O= option is used. So typically, for an out-of-tree build (which are very convenient when using the same Buildroot source tree for different projects/tests), you could do : mkdir ~/myoutput ; make O=~/myoutput menuconfig ; make O=~/myoutput
  • Support for building NPTL toolchains with uClibc, using the latest uClibc snapshots.
  • Support for the gconfig Gtk-based configurator, in addition to the already available menuconfig and xconfig
  • A particular effort has been put on fixing many of the bugs in our Bugzilla, improving robustness thanks to automated random builds, and converting even more packages to the generic and autotools infrastructure
  • Various things have also been deprecated: support for the CRIS, IA64, Sparc64 and Alpha architectures, support for Gtk over DirectFB (which is at the moment not supported upstream), Java support (no maintainer has volunteered to maintain this in Buildroot)
  • Many components have been bumped to newer versions
  • The shared configuration cache, which allowed to speed up the configuration of different packages, has been disabled by default, since it was causing a lot of problems with certain package configurations

I’ve again contributed to a significant portion of this release, being the author of the bootloader build code cleanup, the Linux kernel build code rewrite, leading an effort to reduce the number of outstanding bugs in our Bugzilla and many other little things. The contributors for this release are shown below :

   175  Peter Korsgaard
   168  Thomas Petazzoni
    38  Gustavo Zacarias
    18  cmchao
     8  Luca Ceresoli
     7  Paul Jones
     6  Lionel Landwerlin
     6  Malte Starostik
     5  Yann E. MORIN
     3  Julien Boibessot
     3  Khem Raj
     2  Dmytro Milinevskyy
     2  Francois Perrad
     2  Nick Leverton
     2  Peter Huewe
     2  Stanislav Bogatyrev
     1  Baruch Siach
     1  Bjørn Forsman
     1  Daniel Hobi
     1  Darcy Watkins
     1  Darius Augulis
     1  H Hartley Sweeten
     1  Karl Krach
     1  Kelvin Cheung
     1  Ossy
     1  Sagaert Johan
     1  Simon Pasch
     1  Slava Zanko
     1  Thiago A. Correa
     1  Will Wagner
     1  Yegor Yefremov

For the next release, there are already a few things in the pipeline :

  • Cleanup of all the board support code in Buildroot, in order to cleanly add support for more boards like BeagleBoard, Qemu boards, Calao boards, etc. We’ll use the new minimal defconfig mechanism used by the kernel. I’ve already started working on this
  • Cleanup of the package download process, to support Git and SVN download. The code has already been written by Maxime Petazzoni, reviewed on the list, so I expect it to be included fairly soon
  • Rewrite of libtool handling code, to remove some of our ugly libtool hacks. The code is currently being worked on by Lionel Landwerlin
  • Support for compiling toolchain using Crosstool-NG as a backend. The code is currently being finalized by Yann E. Morin, the author of Crosstool-NG
  • Further work on package uninstallation, clean partial rebuild. Some work has been started by Lionel Landwerlin, but it needs some discussion
  • Continue the conversion of packages to the generic and autotools infrastructures
  • I have also a ton of other things on my TODO-list : rework gdb/gdbserver support with external toolchains, rework the configuration of IPv6/RPC/locale/etc. with external toolchains, set up a Wiki-based Buildroot website with tutorials and better documentation, clean up the toolchain build process, reduce the number of “enhancement” bugs waiting in our Bugzilla, etc.

As Peter Korsgaard, Buildroot maintainer, said in the 2010.08 announcement: The next release is going to be 2010.11. Expect the first release candidate in late October and the final release at the end of November..

It is worth noting that we will be having a Buildroot Developer Day, on Friday 29th October, right after Embedded Linux Conference Europe. At least Peter Korsgaard, Lionel Landwerlin, Yann E. Morin and myself should be there.

Update your WordPress site from scripts

An example Python script, which can be re-used with other website engines.

WordPress logoThe Bootlin website is proudly powered by WordPress. It is a mix of static pages and blog posts, and we are very satisfied of the way we can manage and post content.

Being the webmaster, I had an issue though. Several of our pages share the same bits of content, in particular the descriptions of our public training sessions. To avoid discrepancies between pages, I ended up writing scripts to generate the contents of these pages from common parts. However, updating those pages on the website was done with manual copying and pasting, which was time consuming and error prone.

Fortunately, after some on-line research and practical experiments, I found a simple way of automating the process of login to our WordPress site, open a page for editing, and submitting a new version of the contents.

Since this wasn’t really straightforward, I’m happy to share the update-wordpress-page Python script I came up with, hoping that it be useful to you too.

This script can only be used to modify an existing page. It just changes the contents, and doesn’t touch other attributes. Of course, you could also extend it to create new pages and posts, but this would represent much more work, having lots of input fields to fill.

Another approach would have been to open a direct database connection to the server running WordPress, and then to perform your updates directly with SQL commands. However, this requires a knowledge about WordPress databases (making the script much less generic), and the open database port also makes your website less secure.

To use my script, you will first have to find the edit URL for your page, which reveals the WordPress post id.

I suggest you to create a special WordPress user with Editor privileges. The page history will then show which changes were automated, and which were manual.

The last thing you will have to do is create a $HOME/.update-wordpress-page configuration file as follows:



You can easily tune this script to support other web content engines. You first need to identify the login page (WordPress uses cookies to authenticate a session, instead of simple http authentication). Then, you will find the names of the input forms by reading the login page HTML code.

The second step is to open the page editing URL, and find out the name of the input form used for the page contents.

We don’t offer official support for our script, but I hope that this working code example will help you to make your own scripts, and to get you started faster. Python’s urllib2 and ClientForm really make this easy to do. What I especially like with ClientForm is its ability to modify the value of a given form, without having to read and fill any other input forms in the page, to keep their default content.

Recruiting in Toulouse, France

Penguin worksFor the French speaking readers, we are looking for a graduate engineer to open a new office in Toulouse, France.

All the details are available on our French blog.

This job is not only open to French applicants. Everyone ready to relocate in Toulouse is welcome, but we need someone with a good command of the French language. This will be needed to serve local customers.

ELC Europe 2010 sessions announced

List of sessions and speakers at ELC Europe in Cambridge, UK

Cambridge, UKBeing a member of the organization committee of the Embedded Linux Conference Europe, I get access to fresh news about this yearly conference. The call for presentations is now over and we have just announced the list of sessions.

Note that this list is not the final one yet. Some speakers haven’t confirmed their participation or haven’t sent their biographies yet. There are also two or three speakers added at the last minute who are not listed yet.

The conference will happen in Cambridge, UK, on October 27-28, 2010. Keep an eye on the website (or on our blog). Registration should open in a few days from now, and all practical details will be given then.

See also the agenda of the GStreamer conference which will happen at the same location on the day before.

How to find the root device?

How to find which device corresponds to your root filesystem

I recently found something I was looking for for quite a long time. If you use the mount command in Linux, you can see that the root device is not listed like the other mounted filesystems:

/dev/root on / type ext3 (rw)
/dev/mmcblk0p1 on /mmcboot type vfat (rw)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /dev type tmpfs (rw,mode=0755)

For the / mount point, you are just told that it corresponds to /dev/root, which is not the real device you are looking for.

Of course, you can look at the kernel command line and see on which initial root filesystem Linux was instructed to boot (root parameter):

$ cat /proc/cmdline
mem=512M console=ttyS2,115200n8 root=/dev/mmcblk0p2 rw rootwait

However, this doesn’t mean that what you see is the current root device. Many Linux systems boot on intermediate root filesystems (like initramdisks and initramfs), which are just used to access the final one.

I explored the contents of /proc, but didn’t find any file revealing what the root device is.

Fortunately, I eventually found a command to find the root device:

$ rdev
/dev/mmcblk0p2 /

But how does this work? How could we find such information by ourselves? Use the Source, Luke!

When you ask yourself questions like this one, the best is to look at the BusyBox sources which implement this command. These sources are usually simpler than the ones for the same GNU command.

Here is what BusyBox rdev does… It first runs the stat system call on the / directory. Let’s run the stat command that corresponds to it:

$ stat /
  File: `/'
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: b302h/45826d	Inode: 2           Links: 23
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2010-07-21 22:00:01.000000000 +0200
Modify: 2010-06-13 15:04:37.000000000 +0200
Change: 2010-06-13 15:04:37.000000000 +0200

What’s interesting is the Device field. It means that the device corresponding to / has the major number b3 in hexadecimal (179 in decimal), and minor number 02. Bingo, this corresponds to /dev/mmcblk0p2:

$ ls -l /dev/mmcblk0p2 
brw-rw---- 1 root disk 179, 2 Jan  1  1970 /dev/mmcblk0p2

Therefore, what BusyBox rdev does is walk through /dev and its subdirectories to find a device file matching the major and minor numbers.

This is not a completely generic solution though. On some very simple embedded systems, you don’t even need to create device files for all existing devices. In particular, the device file for the root filesystem doesn’t have to exist. In such a case, rdev wouldn’t be able to find the root device.

A more generic solution could be to walk through /sys/block which enumerates all the block devices present on a system (even if not all of them have an entry in /dev/. This would allow to find the device with the matching major and minor numbers:

$ cat /sys/block/mmcblk0/mmcblk0p1/dev

Through this example, you can see how useful it can be to study the sources of system commands to understand how the system works. BusyBox sources, implementing simplified versions of GNU utilities, make this even easier.

Ubuntu 10.04 on the IGEPv2 board

Installing Ubuntu 10.04 on the IGEPv2 board. Using the board as a small server.

IGEPv2 boardLast year, folks at Texas Instruments told me about the IGEP v2 board. This board is similar to the Beagle board, but also features 512 MB of RAM and NAND flash (instead of 256 for the Beagle), on board Ethernet (RJ45), Wi-Fi and Bluetooth, all this for only 145 €! Its fast ARM CPU (TI OMAP 3530 running at 720 MHz) and graphical capabilities allow it to be used in for services usually performed by desktop or server CPUs.

At the Bootlin main office, we needed a server to share files, create backups and upload these backups to our servers on the Internet. I decided to use the Ubuntu 10.04 distribution on ARM, based on Debian GNU/Linux. As I didn’t find all the details I needed on the IGEP community website, here are the steps that I took. Several details were found on the page though.

This page assumes that you are familiar with building the Linux kernel, controlling an embedded board from a serial line and booting it, and using the GNU/Linux system in general (see the training materials from our embedded Linux course). Beginners may be lost because we don’t give all the details, but more experienced developers should just find the board specific details that they need.

First, get an SD card (at least 2 GB), and prepare its partitions with the utility.

To compile your kernel, get a CodeSourcery toolchain for ARM. I used the 2010q1 release. Install it in /usr/local/CodeSourcery/arm-2010q1/ (for example)

Get the kernel sources:

$ mkdir $HOME/igep
$ cd $HOME/igep
$ git clone git://
$ cd linux-omap-2.6/

Let’s switch to the latest stable version:

$ git tag
$ git checkout -b v2.6.33.4-0 v2.6.33.4-0
Checking out files: 100% (13116/13116), done.
Switched to a new branch 'v2.6.33.4-0'

Now, set the environment variables for cross-compiling the kernel sources to the arm architecture:

$ export PATH=/usr/local/CodeSourcery/arm-2010q1/bin:$PATH
$ export ARCH=arm
$ export CROSS_COMPILE=arm-none-linux-gnueabi-

Now, take the default configuration for the board and build your kernel:

$ make help | grep igep
$ make igep0020_defconfig
$ make -j 4
$ make uImage

It’s time to build your Ubuntu filesystem, using the Rootstock utility:

$ tar zxvf rootstock-
$ cd rootstock-
$ sudo ./rootstock --fqdn igepv2 --login mike --password letmein \
  --imagesize 2G --seed build-essential,openssh-server --dist lucid

Copy the kernel to the first partition of your SD card:

cp arch/arm/boot/uImage /media/boot/
cp .config /media/boot/config-

Install the root filesystem on the second partition of your SD card:

$ cd /media/rootfs/
$ sudo tar zxvf $HOME/igep/rootstock-

Configure the rootfs to let you log in on the serial console (ttyS2 with OMAP). Do this by copying etc/init/tty1.conf to etc/init/ttyS2.conf and replacing tty1 by ttyS2 in this file.

Install kernel modules manually for the first time:

$ mkdir -p /lib/modules
$ cd $HOME/igep/linux-omap-2.6/
$ make INSTALL_MOD_PATH=/media/rootfs modules_install

In the Rootstock version I tested, the specified user didn’t get created (bug report). To be able to log in, I had to disable the root password by removing the first * character in the root entry in etc/shadow:

We are now ready to boot our new system. First, unmount your SD card partitions:

$ sudo umount /media/boot
$ sudo umount /media/rootfs

Insert your SD card in the slot on your board, connect your serial cable and in the U-boot prompt on the serial line, configure the kernel boot parameters:

$ setenv bootargs mem=512M console=ttyS2,115200n8 root=/dev/mmcblk0p2 rw rootwait
$ setenv bootcmd 'mmc init 0 ; fatload mmc 0 80000000 uImage ; bootm 80000000'
$ setenv autostart yes
$ saveenv

You should see your Linux kernel boot and get to a login shell. Log in as root with no password.

It is now time for the final tweaks. First, create a non root user (remember the Rootstock bug), allow it to run the sudo command, and choose a root password too:

adduser mike
adduser mike sudo

Let’s cope with a last Rootstock bug. Add the updates and security repositories to /etc/apt/sources.list:

deb lucid-updates main
deb lucid-security main

Without this, you would miss package updates and security releases, and your packages would never change!

If you use the IGEP board as a server as I do, you may need your server to have a fixed MAC address. The trouble is the e2prom storing the MAC address is not populated by default, and every time you boot, the kernel gives you a random MAC address.

The easiest fix I found was to choose an arbitrary MAC address (you can take the first random one that you get), and force it in /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp
hwaddress ether 00:01:04:1b:2C:1F

As the IGEPv2 board doesn’t have a battery by default, it won’t be able to keep the correct time. You can use the ntp daemon to address this:

sudo apt-get install ntp

Your configuration should now be complete. You can now use your IGEPv2 board as a tiny, ultra low power server with Ubuntu server. All the rest is ordinary Debian / Ubuntu server administration. Of course, you can also install desktop packages and use your board as a desktop replacement (you may need to add kernel command line settings for graphics). Have fun!

By the way, the IGEPv2 board is not the best solution if you all you need is a server. The amazing graphical capabilities of the OMAP chip would just be useless. For a server, better, cheaper and more powerful alternatives are the SheevaPlug and GuruPlug. Don’t miss these very nice devices!

Linux device drivers architecture talk at Libre Software Meeting

recursive device modelThomas Petazzoni gave a talk on the Linux kernel architecture for device drivers at the Libre Software Meeting in Bordeaux, France. While the talk was given in French, the materials are in English and can therefore benefit a larger audience. The talk seems to have been well-received, especially from people already having a basic Linux kernel development experience. The topics covered are part of our Linux Kernel development training, and are also usually very appreciated from the trainees already having Linux kernel experience.

The idea of the talk is to give an overview of how device drivers fit into the kernel, both to expose their functionality to upper layers (such as a network device driver exposes itself to the kernel network infrastructure) and to detect/access the hardware using the device/driver model, which is quite hard to understand from the source code only.

The talk went through the following sections :

  • First a basic introduction to device drivers: how devices are seen from userspace applications, and how a simple, raw, character driver can be implemented. It allowed to expose the principle of operations and their similarity with methods in object-oriented programming, and the principle of registration to an upper-layer infrastructure
  • Then, an introduction to what I call « kernel frameworks », i.e kernel subsystems that specialize a general device type (i.e character device) into a particular device type (i.e serial port device, framebuffer device, etc.). The talk illustrates this with the framebuffer core and the serial port core.
  • Finally, an explanation about the device model: bus drivers, adapter drivers and device drivers. I started with the example of the USB bus: being a dynamically-enumerated bus, it provides a good illustration of the device model principles. At the end, I explained how the device model works for the devices embedded into a SoC using the platform drivers/devices mechanism

The slides for this talk are no longer available, but their updates are now integrated in our Linux kernel and driver development training materials which are freely available.