Location of Embedded Linux Conference Europe 2010 announced

Cambridge, UKWhile the American Embedded Linux Conference always takes place in the San Francisco area in California, the European Embedded Linux Conference Europe travels each year in a different country across Europe: it took place in Austria in 2007, in the Netherlands in 2008 and in France in 2009. The location for the 2010 edition of ELCE has been announced recently: it will take place on October 27th and 28th in Cambridge, United Kingdom. As usual there will be many talks, Bird-of-a-Feather sessions, technical demonstrations and more. If you’re an embedded Linux developer in Europe, you should definitely ask your employer to send you to this conference!

Presentations on the following topics are encouraged: audio, video, and graphics systems for embedded products, security, system size, boot-up time, meeting real-time constraints, power management, streaming media, flash memory devices and filesystems, technologies related to cell phones, digital set top boxes, handheld devices, or other CE products, development tools for embedded users, use of Linux in actual products, practical experience and war stories, standards for CE products. The proposal must be received by June, 30th. My colleague Michael Opdenacker, founder of Bootlin, is a member of the program committee.

On the day before, note that there will also be an interesting event: GStreamer Conference 2010. As GStreamer is a key component of today’s multimedia systems based on embedded Linux. Staying in Cambridge for three days instead of two could then make a lot of sense.

Crosstool-NG 1.7.0, Bootlin contributed basic Blackfin support

Yann E. Morin, maintainer of Crosstool-NG, has just announced version 1.7.0 of his toolchain building tool we like so much at Bootlin. For the record, Crosstool-NG allows to build cross-compiling toolchains for a wide-range of architectures, using either glibc, uClibc, eglibc or newlib, and everything is configured through a menuconfig interface, familiar to all embedded Linux developers.

Blackfin architectureBootlin has contributed to this release, by adding some support for the Blackfin architecture, on which we have been working recently. Crosstool-NG is now capable of generating a FDPIC cross-compiling toolchain for Blackfin. FDPIC allows to use shared libraries even on no-MMU architectures like the Blackfin. This support of Blackfin in Crosstool-NG is only the beginning, a lot of work remains to be done to make sure that the toolchain works really well, and to support FLAT binaries and bare-metal as well.

Here are the patches that Bootlin has contributed for this release :

The other major changes for this release are :

  • two new architectures have been added: Blackfin and mips64
  • building canadian-crosses has been enhanced
  • experimental support for gcc-4.5 has been added
  • eglibc can be optimized for size (instead of speed)
  • companion libraries can be built statically, removing the need for a wrapper
  • many components versions have been added, and associated patchsets updated
  • some less important new features, or enhancements for stability and usability
  • a lot of bug-fixes, of which some to help on Darwin & *BSD

The 1.7.0 release can be downloaded at http://ymorin.is-a-geek.org/download/crosstool-ng/crosstool-ng-1.7.0.tar.bz2 and the homepage of the Crosstool-NG project is http://ymorin.is-a-geek.org/projects/crosstool. Enjoy !

New jobs at Bootlin

Penguin worksLooking for kernel and embedded Linux experts

Bootlin is looking for experienced members of the Free Software community to satisfy increasing demand for development, consulting and training on embedded Linux and on the Linux kernel.

One thing that distinguishes our positions from others is that contributing to the community will be part of your objectives.

All the details can be found on our careers page.

ELCE 2009 videos

Videos from the Embedded Linux Conference Europe, Grenoble, October 2009

ELCE 2009Just a few weeks before the next edition of the Embedded Linux Conference in San Francisco, here are the videos from the previous edition in Europe a few months ago.

These videos were shot by Satoru Ueda and Tim Bird (Sony), Ruud Derwig (NXP) and by Thomas Petazzoni and Michael Opdenacker (Bootlin). As usual, they are released under the terms of the Creative Commons Attribution – ShareAlike Licence version 3.0.

Ruud DerwigIf you have never been to an Embedded Linux Conference yet, these videos should show you how useful this conference is for embedded Linux system developers. This is the place where you can discover new development tools and technologies that will change your working life, benefit from the experience from your peers, get the opportunity to talk to the fantastic people who implement the Free and Open Source software that makes your system run, and win cool penguin goodies. So, don’t miss next next edition in San Francisco.

ELC 2010 program announced

Japantown, San FranciscoThe program of talks and BOFs of the 2010 edition of the Embedded Linux Conference has been published a few days ago, an opportunity to look at the most important and interesting conference for embedded Linux developers. For the record, ELC 2010 will take place from April, 12th to April, 14th in San Francisco, CA, USA, in the same place as the 2009 edition.

A nice set of talks

  • A set of real-time related talks: Real-Time Linux Failure, by Frank Rowand (works for Sony, well known for his preempt-rt related talks at various ELC conferences), Effective Use of RT-Preempt, by Kevin Dankwardt, Using Interrupt Threads to Prioritize Interrupts, by Mike Anderson (also well known for his very interactive talks, he will also be giving his traditional Using JTAG to debug Linux device drivers tutorial), Measuring Responsiveness of Linux Kernel on Embedded System, by YungJoon Jung and DongHyouk Lim.
  • A talk by Grant Likely about Flattened Device Tree ARM support update, an effort to convert the ARM architecture to the same organization used in PowerPC, with a device tree file describing the hardware details instead of platform_device definitions in plain C. An important change for anyone doing ARM kernel development.
  • Several power-management related talks: Runtime Power Management: Overview and Platform Implementation, by Kevin Hillman (who works for Deep Root Systems and has done a huge amount of work in the OMAP power management area). Runtime Power Management is probably the most important change done recently to the power management infrastructure of the Linux kernel, so this talk is certainly worth a look, all the more as Kevin is a very good speaker. On power manegement, there will also be other talks : DVFS for the Embedded Linux, by Yong Bon Koo and Youngbin Seo, Wake-ups effect on idle power for Intel’s Moorestown MID and smartphone platform, by German Monroy (Intel), Workload based aggressive Power Management on the Intel Moorestown MID and future Intel MID/Smartphone Platforms, by Sujith Thomas (from Intel).
  • Japan Town, San FranciscoThe usual tracing-related talks, with Using the LTTng tracer for system-wide performance analysis and debugging by Mathieu Desnoyers and Ftrace – embedded edition, by Steven Rostedt. A talk on debugging Linux toolchain overview with advanced debugging and tracing features, by Dominique Toupin.
  • Talks about platforms: a keynote by Greg Kroah Hartmann on Android: a case study of an embedded Linux project (during which Greg will probably explain why the Android kernel modifications are not mainlined), Experiences in Android Porting, Lessons learned, tips and tricks, by Mark Gross and Understanding and Developing Applications for the Maemo Platform, by Leandro Melo de Sales, even though the recent merge of Maemo and Moblin to create MeeGo is likely to change some technical aspects of application development for this platform.
  • The question of multi-core now also seems to be present in embedded conferences: Strategies for Migrating Uniprocessor Code to Multi-Core, by Mike Anderson, Embedded Multi-core with Adeos, Dan Malek, Lock-free algorithm for Multi-core architecture, Hiromasa Kanda. Multi-core Scheduling optimizations for soft real-time multi-threaded applications – A cooperation aware approach, Lucas Martins De Marchi.
  • Some security talks, with Mike Anderson (again !) talking about Creating a Secure Router Using SELinux and Jake Edge about Understanding threat models for embedded devices
  • Some more-or-less multimedia-oriented talks: Supporting SoC video subsystems in video4linux, by Hans Verkuil, An Introduction to the Qt Development Framework, by Jeremy Katz, GeeXboX Enna: embedded Media Center, by Benjamin Zores, Case Study – Embedded Linux in a digital television STB, by Melanie Rhianna Lewis
  • In the other talks, I’ve noted the Small Business Owners BOF by Grant Likely, Evaluation of Data Reliability on Linux File Systems by Yoshitake Kobayashi, Porting the Linux Kernel to x86 MID platforms, by Jacob Pan, Linux without a bootloader? by Greg Ungerer, Kexec – Ready for Embedded Linux by Magnus Damn, Custom hardware modeling for FPGAs and Embedded Linux Platforms with QEMU, by John Williams, Edgar Iglesias.

Both Michael Opdenacker and I will be there at ELC. We hope to meet you during this conference!

Linux 2.6.33 features for embedded systems

Interesting features for embedded Linux system developers

Penguin workerLinux 2.6.33 was out on Feb. 24, 2010, and to incite you to try this new kernel in your embedded Linux products, here are features you could be interested in.

The first news is the availability of the LZO algorithm for kernel and initramfs compression. Linux 2.6.30 already introduced LZMA and BZIP2 compression options, which could significantly reduce the size of the kernel and initramfs images, but at the cost of much increased decompression time. LZO compression is a nice alternative. Though its compression rate is not as good as that of ZLIB (10 to 15% larger files), decompression time is much faster than with other algorithms. See our benchmarks. We reduced boot time by 200 ms on our at91 arm system, and the savings could even increase with bigger kernels.

This feature was implemented by my colleague Albin Tonnerre. It is currently available on x86 and arm (commit, commit, commit, commit), and according to Russell King, the arm maintainer, it should become the default compression option on this platform. This compressor can also be used on mips, thanks to Wu Zhangjin (commit).

For systems lacking RAM resources, a new useful feature is Compcache, which allows to swap application memory to a compressed cache in RAM. In practise, this technique increases the amount of RAM that applications can use. This could allow your embedded system or your netbook to run applications or environments it couldn’t execute before. This technique can also be a worthy alternative to on-disk swap in servers or desktops which do need a swap partition, as access performance is much improved. See this LWN.net article for details.

This new kernel also carries lots of improvements on embedded platforms, especially on the popular TI OMAP platform. In particular, we noticed early support to the IGEPv2 board, a very attractive platform based on the TI OMAP 3530 processor, much better than the Beagle Board for a very similar price. We have started to use it in customer projects, and we hope to contribute to its full support in the mainline kernel.

Another interesting feature of Linux 2.6.33 is the improvements in the capabilities of the perf tool. In particular, perf probe allows to insert Kprobes probes through the command line. Instead of SystemTap, which relied on kernel modules, perf probe now relies on a sysfs interface to pass probes to the kernel. This means that you no longer need a compiler and kernel headers to produce your probes. This made it difficult to port SystemTap to embedded platforms. The arm architecture doesn’t have performance counters in the mainline kernel yet (other architectures do), but patches are available. This carries the promise to be able to use probe tools like SystemTap at last on embedded architectures, all the more if SystemTap gets ported to this new infrastructure.

Other noticeable improvements in this release are the ability to mount ext3 and ext2 filesystems with just an ext4 driver, a lightweight RCU implementation, as well as the ability to change the default blinking cursor that is shown at boot time.

Unfortunately, each kernel release doesn’t only carry good news. Android patches got dropped from this release, because of a lack of interest from Google to maintain them. These are sad news and a threat for Android users who may end up without the ability to use newer kernel features and releases. Let’s hope that Google will once more realize the value of converging with the mainline Linux community. I hope that key contributors that this company employs (Andrew Morton in particular) will help to solve this issue.

As usual, this was just a selection. You will probably find many other interesting features on the Linux Changes page for Linux 2.6.33.

Embedded Linux practical labs with the Beagle Board

Note: the materials for training with the Beagle Board are no longer available, and would be significantly out of date anyway. We advise you to check our Embedded Linux System Development and Linux Kernel and Driver Development training courses for up-to-date instructions that work on cheaper boards, which are still available on the market today. And if you still have an old Beagle board, it will be an interesting exercise to adapt our current labs to run them on such hardware.

We were asked to customize our embedded Linux training session with specific labs on OMAP 3530 hardware. After a successful delivery on the customer site, using Beagle boards, here are our training materials, released as usual under the terms of the Creative Commons Attribution-ShareAlike 3.0 license:

If you are the happy owner of such a board (both attractive and cheap), or are interested in getting one, you can get valuable embedded Linux experience by reading our lecture materials and by taking our practical labs.

Here’s what you would practise with if you decide to take our labs:

  • Build a cross-compiling toolchain with crosstool-NG
  • Compile U-boot and the X-loader and install it on MMC and flash storage.
  • Manipulate Linux kernel sources and apply source patches
  • Configure, compile and boot a Linux kernel for an emulated PC target
  • Configure, cross-compile and boot a Linux kernel on your Beagle Board
  • Build a tiny filesystem from scratch, based on BusyBox, and with a web server interface. Practice with NFS booting.
  • Put your filesystem on MMC storage, replacing NFS. Practice with SquashFS.
  • Put your filesystem on internal NAND flash storage. Practice with JFFS2 too.
  • Manually cross-compile libraries (zlib, libpng, libjpeg, FreeType and DirectFB) and a DirectFB examples, getting familiar with the tricks required to cross-compile components provided by the community.
  • Build the same kind of graphical system automatically with Buildroot.
  • Compile your own application against existing libraries. Debug a target application with strace, ltrace and gdbserver running on the target.
  • Do experiments with the rt-preempt patches. Measure scheduling latency improvements.
  • Implement hotplugging with mdev, BusyBox’s lightweight alternative to udev.

Note that the labs were tested with Rev. C boards, but are also supposed to work fine with Rev. B ones. You may also be able to reuse some of our instructions with other boards with a TI OMAP3 processor.

Of course, if you like the materials, you can also ask your company to order such a training session from us. We will be delighted to come to your place and spend time with you and your colleagues.

Bootlin at ELCE 2009

Grenoble

As usual, we won’t miss this year’s edition of the Embedded Linux Conference Europe, which has always been a great source of information and encounters for embedded Linux developers.

Here are details about our involvement this year.

  • I am part of the organization committee, in particular the coordinator for the Technical Showcase.
  • Taking advantage of his stay in Grenoble, my colleague Thomas Petazzoni will make an embedded Linux presentation on Tuesday, Oct. 13 at 7:30 pm, at GUILDE, the local Linux user group.
  • Thomas and I will be present at the Embedded Systems Exhibition on Wednesday, Oct. 14, sharing a booth with our partner CALAO Systems. The exhibition entry is free of charge, and this will be an excellent opportunity to meet us and have enough time to talk about your topics of interest.
  • Thomas will lead the Buildroot BOF with Peter Korsgaard, Buildroot’s maintainer, at 5:35 pm on Thursday, Oct. 15. This informal session will allow users and developers to meet and exchange ideas.
  • I will be the leader of the Small Business BOF on Thursday 15 at 6:35 pm, an informal session for small embedded Linux companies interested in sharing experience and best practices, and of course to know each other better.
  • I will make a presentation on boot time reduction techniques, at 3:40 on Friday, Oct. 16.
  • Albin Tonnerre, who was an intern at Bootlin this summer, will participate to the Technical Showcase at 12:00 am on Friday, Oct. 16, showing the benefits of LZO decompression on kernel boot time. During his internship, Albin made very nice contributions to boot time reduction, power management on AT91 and to U-boot board support.
  • Thomas Petazzoni will also participate to the Technical Showcase at the same time, showing Buidroot’s new features.
  • We will videotape the conferences we go to and will release the videos later on our website.
  • Thomas organizes a Buildroot developer day on Saturday, Oct. 17, allowing developers to meet and code together. Bootlin will offer lunch to the participants, and the room will be offered by CALAO Systems. There are no more seats left for space reasons.

Hope to see you in Grenoble!

Faster boot: starting Linux directly from AT91bootstrap

Reducing start-up time looks like one of the most discussed topics nowadays, for both embedded and desktop systems. Typically, the boot process consists of three steps: AT91SAM9263 CPU

  • First-stage bootloader
  • Second-stage bootloader
  • Linux kernel

The first-stage bootloader is often a tiny piece of code whose sole purpose is to bring the hardware in a state where it is able to execute more elaborate programs. On our testing board (CALAO TNY-A9260), it’s a piece of code the CPU stores in internal SRAM and its size is limited to 4Kib, which is a very small amount of space indeed. The second-stage bootloader often provides more advanced features, like downloading the kernel from the network, looking at the contents of the memory, and so on. On our board, this second-stage bootloader is the famous U-Boot.

One way of achieving a faster boot is to simply bypass the second-stage bootloader, and directly boot Linux from the first-stage bootloader. This first-stage bootloader here is AT91bootstrap, which is an open-source bootloader developed by Atmel for their AT91 ARM-based SoCs. While this approach is somewhat static, it’s suitable for production use when the needs are simple (like simply loading a kernel from NAND flash and booting it), and allows to effectively reduce the boot time by not loading U-Boot at all. On our testing board, that saves about 2s.

As we have the source, it’s rather easy to modify AT91bootstrap to suit our needs. To make things easier, we’ll boot using an existing U-Boot uImage. The only requirement is that it should be an uncompressed uImage, like the one automatically generated by make uImage when building the kernel (there’s not much point using such compressed uImage files on ARM anyway, as it is possible to build self-extractible compressed kernels on this platform).

Looking at the (shortened) main.c, the code that actually boots the kernel looks like this:

int main(void)
{
/* ================== 1st step: Hardware Initialization ================= */
/* Performs the hardware initialization */
hw_init();

/* Load from Nandflash in RAM */
load_nandflash(IMG_ADDRESS, IMG_SIZE, JUMP_ADDR);

/* Jump to the Image Address */
return JUMP_ADDR;
}

In the original source code, load_nandflash actually loads the second-stage bootloader, and then jumps directly to JUMP_ADDR (this value can be found in U-Boot as TEXT_BASE, in the board-specific file config.mk. This is the base address from which the program will be executed). Now, if we want to load the kernel directly instead of a second-level bootloader, we need to know a handful of values:

  • the kernel image address (we will reuse IMG_ADDRESS here, but one could
    imagine reading the actual image address from a fixed location in NAND)
  • the kernel size
  • the kernel load address
  • the kernel entry point

The last three values can be extracted from the uImage header. We will not hard-code the kernel size as it was previously the case (using IMG_SIZE), as this would lead to set a maximum size for the image and would force us to copy more data than necessary. All those values are stored as 32 bits bigendian in the header. Looking at the struct image_header declaration from image.h in the uboot-mkimage sources, we can see that the header structure is like this:

typedef struct image_header {
uint32_t    ih_magic;    /* Image Header Magic Number    */
uint32_t    ih_hcrc;    /* Image Header CRC Checksum    */
uint32_t    ih_time;    /* Image Creation Timestamp    */
uint32_t    ih_size;    /* Image Data Size        */
uint32_t    ih_load;    /* Data     Load  Address        */
uint32_t    ih_ep;        /* Entry Point Address        */
uint32_t    ih_dcrc;    /* Image Data CRC Checksum    */
uint8_t        ih_os;        /* Operating System        */
uint8_t        ih_arch;    /* CPU architecture        */
uint8_t        ih_type;    /* Image Type            */
uint8_t        ih_comp;    /* Compression Type        */
uint8_t        ih_name[IH_NMLEN];    /* Image Name        */
} image_header_t;

It’s quite easy to determine where the values we’re looking for actually are in the uImage header.

  • ih_size is the fourth member, hence we can find it at offset 12
  • ih_load and ih_ep are right after ih_size, and therefore can be found at offset 16 and 20.

A first call to load_nandflash is necessary to get those values. As the data we need are contained within the first 32 bytes, that’s all we need to load at first. However, some space is required in memory to actually store the data. The first-stage bootloader is running in internal SRAM, so we can pick any location we want in SDRAM. For the sake of simplicity, we’ll choose PHYS_SDRAM_BASEhere, which we define to the base address of the on-board SDRAM in the CPU address space. Then, a second call will be necessary to load the entire kernel image at the right load address.

Then all we need to do is:

#define be32_to_cpu(a) ((a)[0] << 24 | (a)[1] << 16 | (a)[2] << 8 | (a)[3])
#define PHYS_SDRAM_BASE 0x20000000

int main(void)
{
unsigned char *tmp;
unsigned long jump_addr;
unsigned long load_addr;
unsigned long size;

hw_init();

load_nandflash(IMG_ADDRESS, 0x20, PHYS_SDRAM_BASE);

/* Setup tmp so that we can read the kernel size */
tmp = PHYS_SDRAM_BASE + 12;
size = be32_to_cpu(tmp);

/* Now, load address */
tmp += 4;
load_addr = be32_to_cpu(tmp);

/* And finally, entry point */
tmp += 4;
jump_addr = be32_to_cpu(tmp);

/* Load the actual kernel */
load_nandflash(IMG_ADDRESS, size, load_addr - 0x40);

return jump_addr;
}

Note that the second call to load_nandflash could in theory be replaced by:

load_nandflash(IMG_ADDRESS + 0x40, size + 0x40, load_addr);

However, this will not work. What happens is that load_nandflash starts reading at an address aligned on a page boundary, so even when passing IMG_ADDRESS+0x40 as a first argument, reading will start at IMG_ADDRESS, leading to a failure (writes have to aligned on a page boundary, so it is safe to assume that IMG_ADDRESS is actually correctly aligned).

The above piece of code will silently fail if anything goes wrong, and does no checking at all – indeed, the binary size is very limited and we can’t afford to put more code than what is strictly necessary to boot the kernel.