Busybox 1.18.0, a few new applets

BusyBox logoBusybox, the embedded Linux swiss-army knife, has seen a new release recently: 1.18.0. As usual, it brings dozens of bug fixes, but it also includes a few new applets that are worth noting:

  • pmap, which shows the memory mappings for a particular process. It is just a formatted version of what you can find in /proc/PID/maps, but still nice to have. Knowing the mappings in a particular process is often useful for debugging purposes.
  • add-shell and remove-shell, that respectively add and remove a shell from /etc/shells. This file lists the shells that are acceptable for the chsh command, for example.
  • nandwrite and nanddump, that respectively write to NAND with bad block management, and dumps NAND flash contents. Those two commands expand the set of MTD-related commands in Busybox and are replacements for the version available in mtd-utils. There were already other MTD related utilities in Busybox: flashcp (write to a NOR flash), flash_eraseall (erase a NOR or NAND flash), flash_lock and flash_unlock to protect/unprotect sectors on flash storage. The main advantage is that these utilities relieve you from having to cross-compile mtd-utils, which is never as simple as cross-compiling BusyBox.
  • base64, a base64 encoder and decoder.
  • mpstat, iostat and powertop, three new statistics programs. mpstat (as in multiprocessor stat) reports interrupts and CPU usage on multiprocessor systems, on a per-CPU basis. iostat reports CPU and block device statistics. powertop is a lightweight variant of the popular tool used to diagnose sources of wake-ups, but is only limited to Intel-based machines at the moment.
  • nbd-client, a client for the Network Block Device protocol, so that an embedded system can mount volumes shared by NBD
  • blockdev, allows to perform some ioctl() on block devices such as get sector size, get/set block size, flush buffers, etc.

Have fun with Busybox!

Linux Kernel Development, third edition

Linux Kernel Development, by Robert Love, 3rd edition

Linux Kernel Development is a book authored by Robert Love, a famous kernel developer. Contrary to the very famous Linux Device Drivers book, Linux Kernel Development is not oriented towards driver development, but instead covers how the core Linux kernel works.

Having this knowledge is not absolutely necessary to write Linux device drivers, but having a good overall understanding of the kernel always help to understand what’s going on in your system, even at the application level. In July this year, the third edition of Linux Kernel Development has been published, which upgrades the book contents to kernel version 2.6.34, a good opportunity to have a new look at the book that Bootlin received a few weeks ago.

After a quick introduction to the kernel sources (configuring, building, organization of the source tree), the book immediately dives into kernel internals:

  • Process management: how the kernel represents processes and their state, how processes are created inside the kernel, how threads are handled, are processes are terminated.
  • Process scheduling: a full chapter dedicated to the Linux kernel process scheduler. The new CFS scheduler is of course covered in great detail, with large portion of commented source code, for those who want to understand the fine details of the scheduler. Topics such as process sleep/wake-up, preemption, real-time scheduling policies are also covered.
  • System calls are then covered: how they are implemented, how parameters are passed from userspace to the kernel, etc. The call path from your user-space application down to the kernel is well explained in this chapter.
  • Kernel data structures: a generic chapter which details the kernel API for linked lists, queues, maps, and binary trees. Those APIs are omni-present inside the kernel, and it’s therefore a good idea to know how they work, both for understand existing code and for writing new code.
  • Interrupts: how interrupts are handled and how one can write an interrupt handler. Unfortunately, the newly introduced threaded interrupt handlers are not covered, but it’s true that their usage is not yet very widespread inside the mainline kernel.
  • Bottom halves and deferring work, a topic closely related to interrupt handling. It covers bottom halves, softirqs, tasklets and workqueues.
  • Kernel synchronization: two chapters are dedicated to this topic. First a chapter detailing why synchronization is needed, what are the sources of concurrency and what should be protected against concurrent access. And then a chapter detailing the mechanisms provided by the kernel to implement proper synchronization: atomic operations, spin locks, reader-writer spin locks, semaphores, reader-writer semaphores, mutexes, completion variables, sequential locks, preemption disabling, ordering and barriers
  • Timers and time management details how the kernel manages time: ticks, jiffies counter, timers, delaying execution of code are covered in this chapter. There are unfortunately no details about the clocksource and clockevents infrastructure, and no details about how timers and high-resolution timers are implemented. Contrary to other chapters that go fairly deep into the implementation details, this one mostly only covers the API to time management rather than the internals.
  • Memory management is the topic of the following chapter: physical memory management with the page allocator and the physical zones, then the kmalloc, vmalloc and SLAB allocators are covered. High-memory mappings, a topic specific to 32 bits architectures having more than a gigabyte of RAM is also covered in detail. The per-cpu interface is also covered, and will help those who want to understand parts of the kernel that have been optimized for scalability on multiple CPUs.
  • The Virtual Filesystem, with its different objects: superblock, inode, dentry and file is covered in good detail.
  • The block layer is then covered, with a description of the role of the bio structure, the request queues, and the I/O schedulers.
  • Then, the book goes back to more details about the internals of memory management: the mm_struct memory descriptor, the virtual memory areas (so called VMAs) and how they relate with the mmap/munmap system calls. The next chapter continues with a detailed description of the page cache implementation.
  • A fairly strange chapter called “Devices and modules” gives some information about kernel modules (how to build them, how to use module parameters, how dependencies are handled), then covers the internal of the device model (kobjects, ktypes, ksets, krefs) and finally sysfs. Just like the chapter covering the device model in the third edition of Linux Device Drivers, I think it totally misses the point. All the kobject, ktypes, ksets and krefs stuff is very low-level plumbing used by the Linux device model, but it is not exactly what the driver developer needs to interact with in the first place. In my opinion, a good description of the device model should rather explain what struct bus_type, struct driver and struct device are, how they are specialized for the different bus types in pci_driver, pci_device, usb_driver, usb_device, platform_driver, platform_device, and how the registration/probing of drivers and devices is done. I’ve recently given a talk about this topic, the video is in French, but the slides are in English.
  • Debugging is then covered: printk of course, but also oopses, debugging-related kernel options, the magic SysRq key, kernel debuggers, etc.
  • A rather generic chapter about Portability is then proposed, and finally a chapter about Patches, Hacking and the community that details the kernel community, the kernel coding style, how to generate and submit patches, etc.

All in all, Linux Kernel Development remains very good reading. I particularly appreciate the writing style of Robert Love, who manages to make a deeply-technical book interesting and easy to read. Of course, there are some topics in the kernel in which I had to dive myself and for which I’d expect to see more details in this book, but giving every possible detail about a huge beast such as the Linux kernel in just 400 pages is not possible!

Videos from FOSDEM 2010

Peter Korsgaard presenting Buildroot in the Cross build systems workshop at FOSDEM 2010

Like every year, the Free and Open Source Developer European Meeting took place early February in Brussels, and Thomas Petazzoni, from Bootlin, attended and recorded a few talks from the embedded session. However, contrary to previous years, I haven’t been able to record all talks from the embedded session, since I attended talks from other sessions which were already being recorded by others.

Gian-Carlo Pascutto presenting Embedded software development best practices at FOSDEM 2010

We also attended talks from the X.org and Coreboot developer roooms : videos for the X.org developer room can be found at http://video.fosdem.org/2010/devrooms/xorg/ and videos for the Coreboot developer room can be found at http://video.fosdem.org/2010/devrooms/coreboot/.

Buildroot 2010.08 released!

Buildroot logoOn the last day of August, just in time, the 2010.08 version of Buildroot has been released. For the record, Buildroot is an easy-to-use embedded Linux build system: it can build your toolchain, your root filesystem with all its components (Busybox, libraries, applications, etc.), your kernel and your bootloaders, or any combination of these components.

Amongst the interesting changes in this version :

  • Complete rewrite of the bootloader build code. It contained a lot of legacy, unused and unclear stuff, it is now much easier to use and extend. We’ve removed support for Yaboot and added support for the new Barebox bootloader, and all the code to support AT91Bootstrap, AT91DataFlashBoot, U-Boot, Grub and Grub 2 has been rewritten.
  • Complete rewrite of the Linux kernel build code. It was also complicated to use, with an horribly complicated kernel version selection mechanism, the new code is much easier to configure and use.
  • The configuration file .config is now located in the out-of-tree directory when the O= option is used. So typically, for an out-of-tree build (which are very convenient when using the same Buildroot source tree for different projects/tests), you could do : mkdir ~/myoutput ; make O=~/myoutput menuconfig ; make O=~/myoutput
  • Support for building NPTL toolchains with uClibc, using the latest uClibc snapshots.
  • Support for the gconfig Gtk-based configurator, in addition to the already available menuconfig and xconfig
  • A particular effort has been put on fixing many of the bugs in our Bugzilla, improving robustness thanks to automated random builds, and converting even more packages to the generic and autotools infrastructure
  • Various things have also been deprecated: support for the CRIS, IA64, Sparc64 and Alpha architectures, support for Gtk over DirectFB (which is at the moment not supported upstream), Java support (no maintainer has volunteered to maintain this in Buildroot)
  • Many components have been bumped to newer versions
  • The shared configuration cache, which allowed to speed up the configuration of different packages, has been disabled by default, since it was causing a lot of problems with certain package configurations

I’ve again contributed to a significant portion of this release, being the author of the bootloader build code cleanup, the Linux kernel build code rewrite, leading an effort to reduce the number of outstanding bugs in our Bugzilla and many other little things. The contributors for this release are shown below :

   175  Peter Korsgaard
   168  Thomas Petazzoni
    38  Gustavo Zacarias
    18  cmchao
     8  Luca Ceresoli
     7  Paul Jones
     6  Lionel Landwerlin
     6  Malte Starostik
     5  Yann E. MORIN
     3  Julien Boibessot
     3  Khem Raj
     2  Dmytro Milinevskyy
     2  Francois Perrad
     2  Nick Leverton
     2  Peter Huewe
     2  Stanislav Bogatyrev
     1  Baruch Siach
     1  Bjørn Forsman
     1  Daniel Hobi
     1  Darcy Watkins
     1  Darius Augulis
     1  H Hartley Sweeten
     1  Karl Krach
     1  Kelvin Cheung
     1  Ossy
     1  Sagaert Johan
     1  Simon Pasch
     1  Slava Zanko
     1  Thiago A. Correa
     1  Will Wagner
     1  Yegor Yefremov

For the next release, there are already a few things in the pipeline :

  • Cleanup of all the board support code in Buildroot, in order to cleanly add support for more boards like BeagleBoard, Qemu boards, Calao boards, etc. We’ll use the new minimal defconfig mechanism used by the kernel. I’ve already started working on this
  • Cleanup of the package download process, to support Git and SVN download. The code has already been written by Maxime Petazzoni, reviewed on the list, so I expect it to be included fairly soon
  • Rewrite of libtool handling code, to remove some of our ugly libtool hacks. The code is currently being worked on by Lionel Landwerlin
  • Support for compiling toolchain using Crosstool-NG as a backend. The code is currently being finalized by Yann E. Morin, the author of Crosstool-NG
  • Further work on package uninstallation, clean partial rebuild. Some work has been started by Lionel Landwerlin, but it needs some discussion
  • Continue the conversion of packages to the generic and autotools infrastructures
  • I have also a ton of other things on my TODO-list : rework gdb/gdbserver support with external toolchains, rework the configuration of IPv6/RPC/locale/etc. with external toolchains, set up a Wiki-based Buildroot website with tutorials and better documentation, clean up the toolchain build process, reduce the number of “enhancement” bugs waiting in our Bugzilla, etc.

As Peter Korsgaard, Buildroot maintainer, said in the 2010.08 announcement: The next release is going to be 2010.11. Expect the first release candidate in late October and the final release at the end of November..

It is worth noting that we will be having a Buildroot Developer Day, on Friday 29th October, right after Embedded Linux Conference Europe. At least Peter Korsgaard, Lionel Landwerlin, Yann E. Morin and myself should be there.

Linux device drivers architecture talk at Libre Software Meeting

recursive device modelThomas Petazzoni gave a talk on the Linux kernel architecture for device drivers at the Libre Software Meeting in Bordeaux, France. While the talk was given in French, the materials are in English and can therefore benefit a larger audience. The talk seems to have been well-received, especially from people already having a basic Linux kernel development experience. The topics covered are part of our Linux Kernel development training, and are also usually very appreciated from the trainees already having Linux kernel experience.

The idea of the talk is to give an overview of how device drivers fit into the kernel, both to expose their functionality to upper layers (such as a network device driver exposes itself to the kernel network infrastructure) and to detect/access the hardware using the device/driver model, which is quite hard to understand from the source code only.

The talk went through the following sections :

  • First a basic introduction to device drivers: how devices are seen from userspace applications, and how a simple, raw, character driver can be implemented. It allowed to expose the principle of operations and their similarity with methods in object-oriented programming, and the principle of registration to an upper-layer infrastructure
  • Then, an introduction to what I call « kernel frameworks », i.e kernel subsystems that specialize a general device type (i.e character device) into a particular device type (i.e serial port device, framebuffer device, etc.). The talk illustrates this with the framebuffer core and the serial port core.
  • Finally, an explanation about the device model: bus drivers, adapter drivers and device drivers. I started with the example of the USB bus: being a dynamically-enumerated bus, it provides a good illustration of the device model principles. At the end, I explained how the device model works for the devices embedded into a SoC using the platform drivers/devices mechanism

The slides for this talk are no longer available, but their updates are now integrated in our Linux kernel and driver development training materials which are freely available.