Linux 2.6.30 – New features for embedded systems

Interesting features in Linux 2.6.30 for embedded system developers

Linux 2.6.30 has been released almost 1 month ago and it’s high time to write a little about it. Like every new kernel release, it contains interesting features for embedded system developers.

The first feature that stands out is support for fastboot. Today, most devices are initialized in a sequential way. As scanning and probing the hardware often requires waiting for the devices to be ready, a significant amount of cpu time is wasted in delay loops. The fastboot infrastructure allows to run device initialization routines in parallel, keeping the cpu fully busy and thus reducing boot time in a significant way. Fasboot can be enabled by passing the fastboot parameter in the kernel command line. However, unless your embedded system uses PC hardware, don’t be surprised if you don’t get any boot time reduction yet. If you look at the code, you will see that the async_schedule function is only used by 4 drivers so far, mainly for disk drives. You can see that board support code and most drivers still need to be converted to this new infrastructure. Let’s hope this will progress in future kernel releases, bringing significant boot time savings to everyone.

Tomoyo LinuxLinux 2.6.30 also features the inclusion of Tomoyo Linux, a lightweight Mandatory Access Control system developed by NTT. According to presentations I saw quite a long time ago, Tomoyo can be used as an alternative to SELinux, and it just consumes a very reasonable amount of RAM and storage space. It should interest people making embedded devices which are always connected to the network, and need strong security.

Another nice feature is support for kernel images compressed with bzip2 and lzma, and not just with zlib as it’s been the case of ages. The bzip2 and lzma compressors allow to reduce the size of a compressed kernel in a significant way. This should appeal to everyone interested in saving a few hundreds of kilobytes of storage space. Just beware that decompressing these formats requires more CPU resources, so there may be a price to pay in terms of boot time if you have a slow cpu, like in many embedded systems. On the other hand, if you have a fast cpu and rather slow I/O, like in a PC, you may also see a reduction in boot time. In this case, you would mainly save I/O time copying a smaller kernel to RAM, and with a fast cpu, the extra decompression cost wouldn’t be too high.

However, if you take a closer look at this new feature, you will find that it is only supported on x86 and blackfin. Alain Knaff, the author of the original patches, did summit a patch for the arm architecture, but it didn’t make it this time. Upon my request, Alain posted an update to this arm patch. Unfortunately, decompressing the kernel no longer works after applying this patch. There seems to be something wrong with the decompression code… Stay tuned on the LKML to follow up this issue. Note that the blackfin maintainers took another approach, apparently. They didn’t include any decompression code on this architecture. Instead, they relied on the bootloader to take care of decompression. While this is simpler, at least from the kernel code point of view, this is not a satisfactory solution. It would be best if the arm kernel bootstrap code took care of this task, which would then work with any board and any bootloader.

Another interesting feature is the inclusion of the Microblaze architecture, a soft core cpu on Xilinx FPGAs. This MMU-less core has been supported for quite a long time by uClinux, and it’s good news that it is now supported in the mainline kernel. This guarantees that this cpu will indeed be maintained for a long time, and could thus be a good choice in your designs.

Other noteworthy features are support for threaded interrupt handlers (which shows that work to merge the real-time preempt patches is progressing), ftrace support in 32 and 64 bit powerpc, new tracers, and of course, several new embedded boards and many new device drivers.

As usual, full details can be found on the Linux Kernel Newbies website.

Demo: tiny qemu arm system with a DirectFB interface

A tiny embedded Linux system running on the qemu arm emulator, with a DirectFB interface, everything in 2.1 MB (including the kernel)!

Overview

This demo embedded Linux system has the following features:

  • Very easy to run demo, just 1 file to download and 1 command line to type!
  • Runs on qemu (easy to get for most GNU/Linux distributions), emulating an ARM Versatile PB board.
  • Available through a single file (kernel and root filesystem), sizing only 2.1 MB!
  • DirectFB graphical user interface.
  • Demonstrates the capabilities of qemu, the Linux kernel, BusyBox, DirectFB, and
    shows the benefits of system size and boot time reduction techniques as advertised and supported by the CE Linux Forum.
  • License: GNU GPL for root filesystem scripts. Each software component has its own license.

How to run the demo

  • Make sure the qemu emulator is installed on your GNU/Linux distribution. The demo works with qemu 0.8.2 and beyond, but it may also work with earlier versions.
  • Download the vmlinuz-qemu-arm-2.6.20
    binary.
  • Run the below command:
    qemu-system-arm -M versatilepb -m 16 -kernel vmlinuz-qemu-arm-2.6.20 -append "clocksource=pit quiet rw"
  • When you reach the console prompt, you can try regular Unix commands but also the graphical demo:
    run_demo
    

FAQ / Troubleshooting

  • Q: I get Could not initialize SDL - exiting when I try to run qemu.

    That’s a qemu issue (qemu used the SDL library). Check that you can start graphical applications from your terminal (try xeyes or xterm for example). You may also need to check that you have name servers listed in /etc/resolv.conf. Anyway, you will find solutions for this issue on the Internet.

Screenshots

console screenshot df_andi program screenshot
df_dok program screenshot df_dok2 program screenshot
df_neo program screenshot df_input program screenshot

How to rebuild this demo

All the files needed to rebuild this demo are available here:

  • You can rebuild or upgrade the (Vanilla) kernel by using the given kernel configuration file.
  • The configuration file expects to find an initramfs source directory in ../rootfs, which
    you can create by extracting the contents of the rootfs.tar.7z archive.
  • Of course, you can make changes to this root filesystem!

Tools and optimization techniques used in this demo

Software and development tools

  • The demo was built using Scratchbox, a fantastic development tool that makes cross-compiling transparent!
  • The demo includes BusyBox 1.4.1, an toolbox implementing most UNIX commands in a few hundreds of KB. In our case, BusyBox includes the most common commands (like a vi implementation), and only sizes 192 KB!
  • The root filesystem is shipped within the Linux kernel image, using the initramfs technique, which makes the kernel simpler and saves a dramatic amount of RAM (compared to an init ramdisk).
  • The demo is interfaced by DirectFB example programs (version 0.9.25, with DirectFB 1.0.0-rc4), which demonstrate the amazing capabilities of this library, created to meet the needs of embedded systems.

Size optimization techniques

The below optimization techniques were used to reduce the filesystem size from 74 MB to 3.3 MB (before compression in the Linux kernel image):

  • Removing development files: C headers and manual pages copied when installing tools and libraries, .a library files, gdbserver, strace, /usr/lib/libfakeroot, /usr/local/lib/pkgconfig
  • Files not used by the demo programs: libstdc++, and any library or resource file.
  • Stripping and even super stripping (see sstrip) executables and libraries.
  • Reducing the kernel size using CONFIG_EMBEDDED switches, mainly from the
    Linux Tiny project.

Techniques to reduce boot time

We used the below techniques to reduce boot time:

  • Disabled console output (quiet boot option, printk support was disabled anyway), which saves time scrolling the framebuffer console.
  • Use the Preset Loops per Jiffy technique to disable delay loop calculation, by feeding the kernel with a value measured in an earlier boot (lpj setting, which you may update according to the speed of your own workstation).

All these optimization techniques and other ones we haven’t tried yet are described either on the elinux.org Wiki or in our embedded Linux optimizations presentation.

Future work

We plan to implement a generic tool which would apply some of these techniques in an automatic way, to shrink an existing GNU/Linux or embedded Linux root filesystem without any loss in functionality. More in the next weeks or months!

Conference videos and report

27 free videos from the ELC and FOSDEM 2008 conferences. Extensive technical report from ELC 2008.

After participating to the Embedded Linux Conference (ELC) in Mountain View, and to FOSDEM in Brussels, we are pleased to release the videos that we managed to shoot.

These videos should be useful to anyone interested in the multiple topics covered by these very interesting conferences, either to people who couldn’t join these conferences, or to single core participants who couldn’t attend more than one presentation at once. These videos are also interesting opportunities to see and hear key community members like Andrew Morton, Keith Packard, Henry Kingman, Tim Bird and many others!

While we’ve been releasing free technical videos for a few years now, ELC is the first conference for which we are also offering an extensive report, written by Thomas Petazzoni, one of our kernel and embedded system developers. This report is trying to sum up the most interesting things learned at this conference, at least from the presentations Thomas could attend. This way, you shouldn’t have to view all the videos to identify the most interesting talks.

Creative commons In agreement with the speakers, these videos and the report are released under the terms of the Creative Commons Attribution-ShareAlike 3.0 license.

We hope that sharing this knowledge will attract new contributors and users, and will bring our community one step closer to world domination…

Embedded Linux Conference, Mountain View, Apr. 2008

Don’t miss our detailed report on the below presentations!

  • Keynote: The Relationship Between kernel.org Development and the Use of Linux for Embedded Applications, by Andrew Morton (Google):
    video, slides (55 minutes, 240 MB)
  • UME – Ubuntu Mobile and Embedded, by David Mandala (Canonical):
    video, slides (30 minutes, 145 MB)
  • Appropriate Community Practices: Social and Technical Advice, by Deepak Saxena (MontaVista):
    video (thanks to Kevin Hilman, MontaVista)(44 minutes, 139 MB)
  • Adventures In Real-Time Performance Tuning, by Frank Rowand:
    video,slides (50 minutes, 251 MB)
  • Shifting Sands: Lessons Learned from Linux on an FPGA, by Grant Likely:
    video, slides (44 minutes, 262 MB)
  • Disko – An Application Framework for Digital Media Devices, by Guido Madaus:
    video (27 minutes, 190 MB)
  • Keynote: Tux in Lights, by Henry Kingman (LinuxDevices.com):
    video, slides (44 minutes, 139 MB)
  • Back-tracing in MIPS-based Linux Systems, by Jong-Sung Kim (LG Electronics):
    video, slides
    (54 minutes, 160 MB)
  • Making a Phone Call With Phase Change Memory, by Justin Treon (Numonyx):
    video, slides (28 minutes, 159 MB)
  • Building Blocks for Embedded Power Management, by Kevin Hilman (MontaVista):
    We couldn’t film his presentation, but we already shot a similar presentation he gave at Fosdem 2008: video ((56 minutes, 183 MB)
  • Using Real-Time Linux, by Klaas van Gend (MontaVista):
    video, slides (53 minutes, 263 MB)
  • Every Microamp is Sacred – A Dynamic Voltage and Current Control Interface for the Linux Kernel, by Liam Girdwood (Wolfson Microelectronics):
    video, slides (35 minutes, 71 MB)
  • Power Management Quality of Service and How You Could Use it in Your Embedded Application, by Mark Gross (Intel):
    video, slides (57 minutes, 401 MB)
  • OpenEmbedded for product development, by Matt Locke (Embedded Alley):
    video, slides (49 minutes, 141 MB)
  • Kernel Size Report, and Bloatwatch Update, by Matt Mackall (Selenic Consulting):
    video (49 minutes, 146 MB)
  • Leveraging Free and Open Source Software in a Product Development Environment, by Matt Porter (Embedded Alley):
    video, slides (45 minutes, 220 MB)
  • Using a JTAG for Linux Driver Debugging, by Mike Anderson (PTR Group):
    video, slides (113 minutes, 694 MB)
  • DirectFB Internals – Things You Need to Know to Write Your DirectFB gfxdriver, by Takanari Hayama ():
    video (43 minutes, 200 MB)
  • Linux Tiny – Penguin Weight Watchers, by Thomas Petazzoni (Free Electrons):
    video (thanks to Jean Pihet, MontaVista), slides (32 minutes, 140 MB)
  • Keynote: Status of Embedded Linux and CELF Plenary Meeting, by Tim Bird (Sony):
    video, slides (49 minutes, 112 MB)

Slides are collected on http://www.celinux.org/elc08_presentations/.

Fosdem, Brussels, Feb. 2008

  • Modest, email client for embedded systems, by Dirk-Jan Binnema (Nokia):
    video (34 minutes, 121 MB)
  • Design a Linux robot companion with 8 bits microcontrollers, by David Bourgeois:
    video (54 minutes, 211 MB)
  • Linux on the PS3, by Olivier Grisel:
    video (47 minutes, 272 MB)
  • Xen for Secure Isolation on ARM11, by Jean-Pihet (MontaVista):
    video (41 minutes, 207 MB)
  • Building blocks for Embedded Power Management, by Kevin Hilman (MontaVista):
    video (56 minutes, 183 MB)
  • Emdebian Update: Rootfs, GPE and tdebs, by Neil Williams:
    video (47 minutes, 226 MB)
  • pjsip: lightweight portable SIP stack, by Perry Ismangil:
    video (55 minutes, 194 MB)

Additional video

  • Roadmap to recovery – pain and redemption in X driver development, by Keith Packard:
    video (44 minutes, 168 MB)

ELCE 2007 videos

Free videos of CELF’s Embedded Linux Conference Europe / 9th Real-Time Linux Workshop in Linz, Austria, November 2007.

We are happy to release the videos that we took at the CELF Embedded Linux Conference Europe 2007 / 9th Real-Time Linux Workshop which happened in Linz, Austria in November, 2007.

  • Detection & Resolution of Real Time Issues Using TimeDoctor, by François Audeon (NXP):
    video (32 minutes, 359 MB)
  • Fancy and Fast GUIs on Embedded Devices, by Gustavo Sverzut Barbieri (INDT):
    video, slides (46 minutes, 146 MB)
  • arch/ppc, arch/powerpc and Device Trees – A Walk Through a Port, by Hugh Blemings (IBM):
    video (30 minutes, 534 MB)
  • Free Software, Licensing and Business Processes, by Shane Martin Coughlan (FSF Europe):
    video, slides (40 minutes, 138 MB)
  • Introduction to LogFS, by Jörn Engel:
    video, slides (46 minutes, 260 MB)
  • WebKit on Linux and How It Compares to Other Open Source Engines, by Holger Freyther (Trolltech):
    video, slides (49 minutes, 205 MB)
  • Status Overview of Real-Time, by Thomas Gleixner (Linutronix.de):
    video (47 minutes, 236 MB)
  • Kernel Summit Report, by Thomas Gleixner (Linutronix.de):
    video (34 minutes, 520 MB)
  • Writing DirectFB gfxdriver For Your Embedded System, by Takanari Hayama (igel):
    video, slides (31 minutes, 223 MB)
  • Improving JFFS2 RAM Usage and Performance, by Alexey Korolev (Intel):
    video, slides (20 minutes, 141 MB)
  • YAFFS, by Wookey:
    video, slides (45 minutes, 194 MB)
  • Parallelizing Linux boot on CE Devices, by Vitaly Wool (Embedded Alley Solutions):
    video, slides (40 minutes, 185 MB)
  • Linux Suspend-to-Disk Objectives for Consumer Electronic Devices, by Vitaly Wool (Embedded Alley Solutions):
    video, slides (35 minutes, 652 MB)
  • Evaluation of Linux rt-preempt for embedded industrial devices for Automation and Power technologies – A case study, by Morten Mossige, Pradyumna Sampath, Rachana Rao (ABB):
    video, paper (22 minutes, 224 MB)
  • Assessment of the Realtime Preemption Patches (RT-Preempt) and their impact on the general purpose performance of the system, by Arthur Siro (DSLab / OSADL):
    video, paper (31 minutes, 224 MB)
  • Panel: the ideal embedded Linux distribution, by Tim Bird (Sony):
    video (65 minutes, 465 MB)

To speed up the processing of these videos, we contracted Jan Gerber, the developer of ffmpeg2theora, to add denoising support to this tool. Thanks to this contribution, it is now possible for anyone in the community to directly denoise DV camcorder input and generate Ogg/Theora video in just 1 step. Before it was necessary to use mencoder‘s denoising filter, and because mencoder couldn’t process DV input properly, a preprocessing stage with ffmpeg was also required. This new functionality can also improve the quality and compression rate of live Ogg/Theora video broadcasts.

Embedded Linux and Ecology

Embedded Linux contributions to the Linux Ecology HOWTO.

Free Electrons has contributed major updates to the Linux Ecology HOWTO, a Linux Documentation Project document that gathers ideas and techniques for using Linux in an environmentally friendly way.

In particular, Free Electrons took advantage of its experience with embedded Linux system development to add new techniques which can reduce power consumption or make it possible to extend the lifetime of old systems with limited resources.

Free Electrons also contributed an overview presentation on this HOWTO. The latest HOWTO version with our updates (waiting for the next official release) can also be found on the same page.

CELF conference videos

Videos from the 2006 CE Linux Forum conference in Santa Clara, California

For people who couldn’t make it to the 2006 edition of the CE Linux Forum conference, Free Electrons is happy to share the following videos, shot by Michael Opdenacker:

  • Keynote, by Tim Bird (Sony):
    video, slides (46 minutes, 139 MB)
  • Usability of User-Space device drivers, by a speaker from Renesas:
    video, slides (50 minutes, 256 MB)
  • What’s New with Busybox, by Rob Landley (TimeSys):
    video (55 minutes, 322 MB)
  • Topics in Embedded Power Management, by Todd Poynor (MontaVista):
    video, slides (44 minutes, 253 MB)
  • CE Linux Forum Open Test Lab, by Matt Locke (Nomad Global Solutions):
    video (35 minutes, 213 MB)
  • Power Management Panel, by Mark Gross (Intel):
    video (55 minutes, 360 MB)
  • Panel: Tips for Mainlining, From the Experts (starring Greg K.H., Greg Ungerer, Matt Mackall), by Tim Bird (Sony):
    video (34 minutes, 230 MB)
  • MIPS BOF, by Arvind Kumar (MIPS):
    video (53 minutes, 268 MB)
  • uClinux, by Greg Ungerer (SnapGear):
    video, slides (44 minutes, 235 MB)
  • Examining Linux Kernel Size, by Munehiro Ikeda (NEC):
    video, slides (42 minutes, 247 MB)
  • Integrating DirectFB into a UHAPI Platform, by Denis Kropp (directfb.org):
    video, slides (31 minutes, 93 MB)
  • Linux Trace Toolkit Next Generation, by Mathieu Desnoyers (École Polytechnique de Montréal):
    video, slides (34 minutes, 211 MB)
  • MythTV on Philips Nexperia PNX8550, by Klaas de Waal (Philips):
    video, slides (50 minutes, 293 MB)
  • Visualizing Resource Usage During Initialization of Embedded Systems (embootchart), by Matthew Klahn (Motorola):
    video, slides (51 minutes, 236 MB)
  • State of Linux Real-time BOF, by Manas Saksena (TimeSys):
    video (42 minutes, 297 MB)
  • Closing Activity: Embedded Linux Quiz, by Tim Bird (Sony):
    video (22 minutes, 235 MB)

Slides are being collected on http://tree.celinuxforum.org/CelfPubWiki/ELC2006Presentations.

You may also be interested in our report, published by LWN.net.

Fosdem 2006 videos

Videos from the Fosdem conference in Brussels

For people who couldn’t go to the 2006 edition of Fosdem, one of the 2 most popular worldwide meetings for Free Software developers, or who couldn’t attend all the presentations they were interested in, Free Electrons is happy to share the below 12 videos from this event:

  • Software patents, by Richard Stallman (Free Software Foundation):
    video (83 minutes, 345 MB)
  • SETR LiveCD, live tool for embedded development, by Hector Oron:
    video (30 minutes – lacks the first 5 minutes, 93 MB)
  • Optimizing the Linux kernel and applications, by
    Michael Opdenacker (Free Electrons): video (49 minutes, 129 MB)
  • Lock-Free Data Exchange for Real-time applications, by Peter Soetens (Flander’s Mechatronics Technology Center, Leuven):
    video (49 minutes, 170 MB)
  • Alsa SoC layer, by Liam Girdwood (Wolfson Microelectronics):
    video (37 minutes, 95 MB)
  • The Maemo development platform, by Tommi Komulainen (Nokia):
    video (45 minutes, 151 MB)
  • Building and supporting distributions with OpenEmbedded,
    by Koen Kooi (Familiar release manager):
    video (48 minutes, 179 MB)
  • Towards a common embedded power management solution, by
    David Weinehall and Sampsa Fabritius (Nokia): video (81 minutes, 292 MB)
  • Real-time patches for Linux, by Klaas van Gend (MontaVista):
    video (55 minutes, 126 MB)
  • SLIND, a small Debian for embedded systems, by Benjamin Collar (Siemens):
    video (55 minutes, 161 MB)
  • Openocd, the Open On-Chip Debugger, by Dominic Rath (University of Applied Sciences, Augsburg):
    video (40 minutes, 128 MB)
  • The challenge of the GNU/Linux desktop, by Jeff Waugh (Ubuntu / Gnome):
    video (53 minutes, 176 MB)

See the abstracts of the presentations in the embedded track. Many other Fosdem videos can be found here.

Though most videos are related to the embedded topic, anyone could be be interested in Richard Stallman’s arguments on Software Patents as well as in Jeff Vaugh’s very entertaining keynote.

These videos are available under the Creative Commons Attribution – Share Alike 2.5 license.

Many other videos from Fosdem can be found here.

Free embedded Linux training materials

Free Electrons embedded Linux training materials freely available

This was our first, initial annoucement in 2004. Since then, we have made huge improvements to our embedded Linux and Linux kernel and device driver development training courses. See all our training materials.

The 500 page materials of Free Electrons’ embedded Linux training have just been published.

They are all released under the terms of the “GNU Free Documentation License (with no invariant sections).

Full training materials

Presentations