New jobs at Bootlin

Penguin worksLooking for kernel and embedded Linux experts

Bootlin is looking for experienced members of the Free Software community to satisfy increasing demand for development, consulting and training on embedded Linux and on the Linux kernel.

One thing that distinguishes our positions from others is that contributing to the community will be part of your objectives.

All the details can be found on our careers page.

Linux 2.6.33 features for embedded systems

Interesting features for embedded Linux system developers

Penguin workerLinux 2.6.33 was out on Feb. 24, 2010, and to incite you to try this new kernel in your embedded Linux products, here are features you could be interested in.

The first news is the availability of the LZO algorithm for kernel and initramfs compression. Linux 2.6.30 already introduced LZMA and BZIP2 compression options, which could significantly reduce the size of the kernel and initramfs images, but at the cost of much increased decompression time. LZO compression is a nice alternative. Though its compression rate is not as good as that of ZLIB (10 to 15% larger files), decompression time is much faster than with other algorithms. See our benchmarks. We reduced boot time by 200 ms on our at91 arm system, and the savings could even increase with bigger kernels.

This feature was implemented by my colleague Albin Tonnerre. It is currently available on x86 and arm (commit, commit, commit, commit), and according to Russell King, the arm maintainer, it should become the default compression option on this platform. This compressor can also be used on mips, thanks to Wu Zhangjin (commit).

For systems lacking RAM resources, a new useful feature is Compcache, which allows to swap application memory to a compressed cache in RAM. In practise, this technique increases the amount of RAM that applications can use. This could allow your embedded system or your netbook to run applications or environments it couldn’t execute before. This technique can also be a worthy alternative to on-disk swap in servers or desktops which do need a swap partition, as access performance is much improved. See this LWN.net article for details.

This new kernel also carries lots of improvements on embedded platforms, especially on the popular TI OMAP platform. In particular, we noticed early support to the IGEPv2 board, a very attractive platform based on the TI OMAP 3530 processor, much better than the Beagle Board for a very similar price. We have started to use it in customer projects, and we hope to contribute to its full support in the mainline kernel.

Another interesting feature of Linux 2.6.33 is the improvements in the capabilities of the perf tool. In particular, perf probe allows to insert Kprobes probes through the command line. Instead of SystemTap, which relied on kernel modules, perf probe now relies on a sysfs interface to pass probes to the kernel. This means that you no longer need a compiler and kernel headers to produce your probes. This made it difficult to port SystemTap to embedded platforms. The arm architecture doesn’t have performance counters in the mainline kernel yet (other architectures do), but patches are available. This carries the promise to be able to use probe tools like SystemTap at last on embedded architectures, all the more if SystemTap gets ported to this new infrastructure.

Other noticeable improvements in this release are the ability to mount ext3 and ext2 filesystems with just an ext4 driver, a lightweight RCU implementation, as well as the ability to change the default blinking cursor that is shown at boot time.

Unfortunately, each kernel release doesn’t only carry good news. Android patches got dropped from this release, because of a lack of interest from Google to maintain them. These are sad news and a threat for Android users who may end up without the ability to use newer kernel features and releases. Let’s hope that Google will once more realize the value of converging with the mainline Linux community. I hope that key contributors that this company employs (Andrew Morton in particular) will help to solve this issue.

As usual, this was just a selection. You will probably find many other interesting features on the Linux Changes page for Linux 2.6.33.

LZO kernel compression

As Michael stated in his review of the interesting features in Linux 2.6.30, new compression options have been recently added to the kernel. We therefore decided to have a look at those compression methods, from a compression ratio and decompression speed point of view.

This comparison will be based on “self-extractible kernels”, that is, kernel images containing bootstrap code allowing them to extract a compressed image. As underlined in the previous article, this approach is not used on all architectures. Blackfin, notably, chose a different path and compresses the whole kernel image, without including bootstrap code. While this has the clear advantage of making compression much simpler with respect to kernel code, it forces decompression out to the bootloader code.

Each of those methods has its advantages. Indeed, the Blackfin approach relies on the bootloader to provide the necessary functions, so that may be a problem to do things like bypassing u-boot to reduce the boot time. On the other hand, implementing it only once in the bootloader (as architecture-independent code) makes it unnecessary to write the low-level bootstrap code for each architecture in the kernel, which is surely interesting on virtually all architectures, the notable exceptions being x86 and ARM.

Gzip (also known as Zlib or inflate) has been the traditional (and, as a matter of fact, only) method used to compress kernels. Consequently, we’ll use it as the reference in the following tests. Our test environment is as follows:

ARM9 AT91SAM9263 CPU, 200MHz, using the mainline arch/arm/configs/usb-a9263.config.

This comparison includes figures for LZO, a new kernel image compression method that I have contributed to the Linux sources, and which hopefully will make its way into the mainline kernel. LZO support in the kernel is only new for kernel decompression, as it is already used by JFFS2 and UBIFS. LZO is a stream-oriented algorithm, and although its compression ratio is lower than that of gzip, decompression is lightning-fast, as we will soon find out.

So, here are the figures, average on 20 boots with each compression method:

Uncompressed 3.24Mo 200%
LZO 1.76Mo 0.552s 70% 109%
Gzip 1.62Mo 0.775s 100% 100%
LZMA 1.22Mo 5.011s 646% 75%

Bzip2 has not been tested here: the low-level bootstrap file, head.S, only allocates 64Kb for use by malloc() on ARM. Some quick tests showed that the kernel would not extract with less than 3.5Mib of malloc() space. That would require to modify head.S so that malloc can use more memory, which we will not do here. However, given that enough memory is usable on the system, one could well use bzip2. All the other algorithms performed the extraction using the standard 64Kib malloc space.

From the results, we can clearly see that LZMA is nearly unsuitable for our system, and should be considered only if the space constraints for storing the kernel are so tight that we can’t afford to use more space that was is strictly necessary.

LZO looks like a good candidate when it comes to speeding up the boot process, at the expense of some (almost neglectable) extra space. Gzip is close to LZO when it comes to size, although extraction is not as fast. That means that unless you’re hitting corner cases, like only having enough space for a Gzip compressed image but not for one made with LZO, choosing the latter is probably a safe bet.

Besides, the LZO-compressed kernel size is about 54% the size of the uncompressed kernel. As the kernel load time varies linearly with its size, load time for an uncompressed kernel doubles. While 0.55s are won because there’s no need to run a decompression algorithm, you spend twice as much time loading the kernel. This time is not negligible at all compared to the decompression time. Indeed, loading the uncompressed image takes roughly 0.8s. That means that at the cost of slowing down the boot process by 0.15s (compared to an uncompressed kernel), one gets a kernel image which is roughly twice as small. Rather nice, isn’t it?

Faster boot: starting Linux directly from AT91bootstrap

Reducing start-up time looks like one of the most discussed topics nowadays, for both embedded and desktop systems. Typically, the boot process consists of three steps: AT91SAM9263 CPU

  • First-stage bootloader
  • Second-stage bootloader
  • Linux kernel

The first-stage bootloader is often a tiny piece of code whose sole purpose is to bring the hardware in a state where it is able to execute more elaborate programs. On our testing board (CALAO TNY-A9260), it’s a piece of code the CPU stores in internal SRAM and its size is limited to 4Kib, which is a very small amount of space indeed. The second-stage bootloader often provides more advanced features, like downloading the kernel from the network, looking at the contents of the memory, and so on. On our board, this second-stage bootloader is the famous U-Boot.

One way of achieving a faster boot is to simply bypass the second-stage bootloader, and directly boot Linux from the first-stage bootloader. This first-stage bootloader here is AT91bootstrap, which is an open-source bootloader developed by Atmel for their AT91 ARM-based SoCs. While this approach is somewhat static, it’s suitable for production use when the needs are simple (like simply loading a kernel from NAND flash and booting it), and allows to effectively reduce the boot time by not loading U-Boot at all. On our testing board, that saves about 2s.

As we have the source, it’s rather easy to modify AT91bootstrap to suit our needs. To make things easier, we’ll boot using an existing U-Boot uImage. The only requirement is that it should be an uncompressed uImage, like the one automatically generated by make uImage when building the kernel (there’s not much point using such compressed uImage files on ARM anyway, as it is possible to build self-extractible compressed kernels on this platform).

Looking at the (shortened) main.c, the code that actually boots the kernel looks like this:

int main(void)
{
/* ================== 1st step: Hardware Initialization ================= */
/* Performs the hardware initialization */
hw_init();

/* Load from Nandflash in RAM */
load_nandflash(IMG_ADDRESS, IMG_SIZE, JUMP_ADDR);

/* Jump to the Image Address */
return JUMP_ADDR;
}

In the original source code, load_nandflash actually loads the second-stage bootloader, and then jumps directly to JUMP_ADDR (this value can be found in U-Boot as TEXT_BASE, in the board-specific file config.mk. This is the base address from which the program will be executed). Now, if we want to load the kernel directly instead of a second-level bootloader, we need to know a handful of values:

  • the kernel image address (we will reuse IMG_ADDRESS here, but one could
    imagine reading the actual image address from a fixed location in NAND)
  • the kernel size
  • the kernel load address
  • the kernel entry point

The last three values can be extracted from the uImage header. We will not hard-code the kernel size as it was previously the case (using IMG_SIZE), as this would lead to set a maximum size for the image and would force us to copy more data than necessary. All those values are stored as 32 bits bigendian in the header. Looking at the struct image_header declaration from image.h in the uboot-mkimage sources, we can see that the header structure is like this:

typedef struct image_header {
uint32_t    ih_magic;    /* Image Header Magic Number    */
uint32_t    ih_hcrc;    /* Image Header CRC Checksum    */
uint32_t    ih_time;    /* Image Creation Timestamp    */
uint32_t    ih_size;    /* Image Data Size        */
uint32_t    ih_load;    /* Data     Load  Address        */
uint32_t    ih_ep;        /* Entry Point Address        */
uint32_t    ih_dcrc;    /* Image Data CRC Checksum    */
uint8_t        ih_os;        /* Operating System        */
uint8_t        ih_arch;    /* CPU architecture        */
uint8_t        ih_type;    /* Image Type            */
uint8_t        ih_comp;    /* Compression Type        */
uint8_t        ih_name[IH_NMLEN];    /* Image Name        */
} image_header_t;

It’s quite easy to determine where the values we’re looking for actually are in the uImage header.

  • ih_size is the fourth member, hence we can find it at offset 12
  • ih_load and ih_ep are right after ih_size, and therefore can be found at offset 16 and 20.

A first call to load_nandflash is necessary to get those values. As the data we need are contained within the first 32 bytes, that’s all we need to load at first. However, some space is required in memory to actually store the data. The first-stage bootloader is running in internal SRAM, so we can pick any location we want in SDRAM. For the sake of simplicity, we’ll choose PHYS_SDRAM_BASEhere, which we define to the base address of the on-board SDRAM in the CPU address space. Then, a second call will be necessary to load the entire kernel image at the right load address.

Then all we need to do is:

#define be32_to_cpu(a) ((a)[0] << 24 | (a)[1] << 16 | (a)[2] << 8 | (a)[3])
#define PHYS_SDRAM_BASE 0x20000000

int main(void)
{
unsigned char *tmp;
unsigned long jump_addr;
unsigned long load_addr;
unsigned long size;

hw_init();

load_nandflash(IMG_ADDRESS, 0x20, PHYS_SDRAM_BASE);

/* Setup tmp so that we can read the kernel size */
tmp = PHYS_SDRAM_BASE + 12;
size = be32_to_cpu(tmp);

/* Now, load address */
tmp += 4;
load_addr = be32_to_cpu(tmp);

/* And finally, entry point */
tmp += 4;
jump_addr = be32_to_cpu(tmp);

/* Load the actual kernel */
load_nandflash(IMG_ADDRESS, size, load_addr - 0x40);

return jump_addr;
}

Note that the second call to load_nandflash could in theory be replaced by:

load_nandflash(IMG_ADDRESS + 0x40, size + 0x40, load_addr);

However, this will not work. What happens is that load_nandflash starts reading at an address aligned on a page boundary, so even when passing IMG_ADDRESS+0x40 as a first argument, reading will start at IMG_ADDRESS, leading to a failure (writes have to aligned on a page boundary, so it is safe to assume that IMG_ADDRESS is actually correctly aligned).

The above piece of code will silently fail if anything goes wrong, and does no checking at all – indeed, the binary size is very limited and we can’t afford to put more code than what is strictly necessary to boot the kernel.

Linux 2.6.30 – New features for embedded systems

Interesting features in Linux 2.6.30 for embedded system developers

Linux 2.6.30 has been released almost 1 month ago and it’s high time to write a little about it. Like every new kernel release, it contains interesting features for embedded system developers.

The first feature that stands out is support for fastboot. Today, most devices are initialized in a sequential way. As scanning and probing the hardware often requires waiting for the devices to be ready, a significant amount of cpu time is wasted in delay loops. The fastboot infrastructure allows to run device initialization routines in parallel, keeping the cpu fully busy and thus reducing boot time in a significant way. Fasboot can be enabled by passing the fastboot parameter in the kernel command line. However, unless your embedded system uses PC hardware, don’t be surprised if you don’t get any boot time reduction yet. If you look at the code, you will see that the async_schedule function is only used by 4 drivers so far, mainly for disk drives. You can see that board support code and most drivers still need to be converted to this new infrastructure. Let’s hope this will progress in future kernel releases, bringing significant boot time savings to everyone.

Tomoyo LinuxLinux 2.6.30 also features the inclusion of Tomoyo Linux, a lightweight Mandatory Access Control system developed by NTT. According to presentations I saw quite a long time ago, Tomoyo can be used as an alternative to SELinux, and it just consumes a very reasonable amount of RAM and storage space. It should interest people making embedded devices which are always connected to the network, and need strong security.

Another nice feature is support for kernel images compressed with bzip2 and lzma, and not just with zlib as it’s been the case of ages. The bzip2 and lzma compressors allow to reduce the size of a compressed kernel in a significant way. This should appeal to everyone interested in saving a few hundreds of kilobytes of storage space. Just beware that decompressing these formats requires more CPU resources, so there may be a price to pay in terms of boot time if you have a slow cpu, like in many embedded systems. On the other hand, if you have a fast cpu and rather slow I/O, like in a PC, you may also see a reduction in boot time. In this case, you would mainly save I/O time copying a smaller kernel to RAM, and with a fast cpu, the extra decompression cost wouldn’t be too high.

However, if you take a closer look at this new feature, you will find that it is only supported on x86 and blackfin. Alain Knaff, the author of the original patches, did summit a patch for the arm architecture, but it didn’t make it this time. Upon my request, Alain posted an update to this arm patch. Unfortunately, decompressing the kernel no longer works after applying this patch. There seems to be something wrong with the decompression code… Stay tuned on the LKML to follow up this issue. Note that the blackfin maintainers took another approach, apparently. They didn’t include any decompression code on this architecture. Instead, they relied on the bootloader to take care of decompression. While this is simpler, at least from the kernel code point of view, this is not a satisfactory solution. It would be best if the arm kernel bootstrap code took care of this task, which would then work with any board and any bootloader.

Another interesting feature is the inclusion of the Microblaze architecture, a soft core cpu on Xilinx FPGAs. This MMU-less core has been supported for quite a long time by uClinux, and it’s good news that it is now supported in the mainline kernel. This guarantees that this cpu will indeed be maintained for a long time, and could thus be a good choice in your designs.

Other noteworthy features are support for threaded interrupt handlers (which shows that work to merge the real-time preempt patches is progressing), ftrace support in 32 and 64 bit powerpc, new tracers, and of course, several new embedded boards and many new device drivers.

As usual, full details can be found on the Linux Kernel Newbies website.

Linux kernel 2.6.29 – New features for embedded users

Tuz Linux logoThe 2.6.29 version of the Linux kernel has just been released by Linus Torvalds. Like all kernel releases, this new version offers a number of interesting new features.

For embedded users, the most important new feature is certainly the inclusion of Squashfs, a read-only compressed filesystem. This filesystem is very well-suited to store the immutable parts of an embedded system (applications, libraries, static data, etc.), and replaces the old cramfs filesystem which had some strong limitations (file size, filesystem size and limited compression).

Squashfs is a block filesystem, but since it is read-only, you can also use it on flash partitions, through the mtdblock driver. It’s fine as you just write the filesystem image once. Don’t hesitate to try it to get the best performance out of your flash partitions. Ideally, you should even use it on top of UBI, which would transparently allow the read-only parts of your filesystem to participate to wear-leveling. See sections about filesystems in our embedded Linux training materials for details.

This new release also adds Fastboot support, at least for scsi probes and libata port scan. This is a step forward to reducing boot time, which is often critical in embedded systems.

2.6.29 also allows stripping of generated symbols under CONFIG_KALLSYMS_ALL, saving up to 200 KB for kernels built with this feature. This is nice for embedded systems with very little RAM, which still need this feature during development.

Another new feature of this kernel is the support for the Samsung S3C64XX CPUs. Of course, a lot of new boards and devices are supported, such as the framebuffer of the i.MX 31 CPU, or the SMSC LAN911x and LAN912x Ethernet controllers.

The other major features of this new kernel, not necessarily interesting for embedded systems are the inclusion of the btrfs filesystem, the inclusion of the kernel modesetting code, support for WiMAX, scalability improvements (support of 4096 CPUs!), filesystem freezing capability, filesystem improvements and many new drivers.

For more details on the 2.6.29 improvements, the best resource is certainly the human-readable changelog written by the KernelNewbies.org community.

Kernel and BSP development and mainlining

Offering free mainlining for Linux kernel, device driver and BSP development

Note: Due to the number of requests we get for mainlining work, we can no longer continue the offer described below.

Bootlin is best known worldwide for its kernel and embedded Linux system training sessions and its free training materials, and also perhaps for sharing videos from technical conferences.

However, did you know that Bootlin is not a training company?

We are actually embedded Linux system and kernel developers like the people we support, train and work for. This is essential to be good trainers, in addition to our passion for sharing what we learn. For example, have you ever had a look at the description of our engineering services?

In our training activity, we differentiate with other suppliers by offering custom sessions, being completely transparent with our training materials, costs and customer evaluations. Here’s how we can also make a difference in our development activity:

  • As in all our activities, by focusing only on the Linux kernel, device drivers, bootloaders, embedded and real-time system development.
  • By offering free mainlining to any customer who orders Linux board support packages or Linux device drivers from us. Having our code accepted in the official Linux and U-boot sources brings terrific benefits to our customers and can be a key contributor to the success of their products.
  • By working inside the Linux development community. Being part of it, we know this community very well: its people, its rules, its best practices and its best resources. This helps us to make the right decisions (if needed, collecting advice from the right people), and quickly obtain the expected results.

2008 contributions

Here are the contributions that we made to the user and developer community in 2008, thanks to the customers who ordered our engineering and training services.

As you can see, we do our best to have all our contributions merged into mainline sources. So, if you need a new feature in the Linux kernel (supporting your new boards, for example), in development tools and libraries (Buildroot, QEMU…), and want to enjoy this feature in all future updates and releases, don’t hesitate to ask us. We will be glad to work with the community and find a long lasting solution.

Linux kernel

  • [x86] use ELF section to list CPU vendor specific code (commit)
  • [MTD] fix minor typo in the MTD map driver for SHARP SL series (commit)
  • [x86] configurable DMI scanning code (commit)
  • [mm] directly use kmalloc() and kfree() in init/initramfs.c (commit)
  • [x86] consolidate the definition of the force_mwait variable (commit)
  • inflate: refactor inflate malloc code (commit)
  • fs/buffer.c: uninline __remove_assoc_queue() (commit)
  • [x86] make movsl_mask definition non-CPU specific (commit).
  • [x86] move cmpxchg fallbacks to a generic place (commit)
  • [x86] configuration options to compile out x86 CPU support code (commit)
  • Configure out file locking features (commit)
  • Fix comment in include/linux/mmc/host.h (commit)
  • Configure out AIO support (commit)
  • [PCI] allow quirks to be compiled out (commit)
  • [x86] remove PC speaker code (commit)
  • [Doc] improvement to Documentation/SubmittingPatches (commit)
  • Work on multicast and ethtool configurability. Not merged yet.
  • 65 e-mails sent to the kernel newbies mailing list to help new kernel developers.

Buildroot

Thomas Petazzoni became official committer in November 2008. In addition to contributions, patch review and integration of patches into the official Buildroot repository, and discussions on the mailing list.

  • Thumb support, not integrated yet (post)
  • Fixed URL for fakeroot sources, integrated (post)
  • Bumped libpng version, integrated (post)
  • Added the DirectFB examples package, integrated (post)
  • Bumped up libgtk2 version, integrated (post)
  • Work on external toolchain support, integrated. Several iterations, patches and discussions.
  • External toolchain support improvements, integrated (post)
  • More external toolchain fixes, integrated (post)
  • External toolchain C++ cross-compiler fix, integrated (post)
  • Kernel build fix related to external toolchain use, integrated (post)
  • Fixed external toolchain build, integrated, and replaced later with an improved version (post)
  • Fix Qtopia build issues, integrated and then replaced by an improved version (post)
  • Another external toolchain support solution, integrated (post)
  • Fixed TARGET_PATH for external toolchain builds, integrated (post)
  • Fixed strange problems in pango configure target, integrated (post)
  • Strip libgtk2 in the target, integrated (post)
  • Strip pango libraries on the target, integrated (post)
  • Strip gettext libraries on the target, integrated (post)
  • Fix matchbox build, integrated (post)
  • Create zlib installation directory in the staging dir, integrated (post)
  • Bump up lite version, integrated (post)
  • Added a parallel compilation fix for fontconfig, integrated (post)
  • Documentation fixes (post, post)

QEMU

  • Increased write buffer size in pflash emulation, integrated (post)
  • Reset wcycle after erase confirm, integrated (post)
  • Improved pflash cfi01 debug messages, integrated (post)
  • Added missing parenthesis in qemu_ram_alloc(), integrated (post)
  • Add Flash support to the Versatile PB platform (post)

Conferences

New training materials

All our training materials can now be found on our docs page. Some of them are not new, but have undergone substantial updates.

See our recent post for details.

Financial support

We support organizations promoting Free Software:

You may also count our subscriptions to the most useful LWN.net resource.

Miscellaneous

Kernel 2.6.28 is out with a few Bootlin contributions

A few hours before Christmas, Linus Torvalds released the latest stable version of the Linux kernel, 2.6.28. Jake Edge from LWN sums up the major highlights of this new release: « Some of the highlights of this kernel are the addition of the GEM GPU memory manager, the ext4 filesystem is no longer “experimental”, scalability improvements in memory management via the reworked vmap() and pageout scalability patches, moving the -staging drivers into the mainline, and much more ». As usual, the Kernel Newbies website offers an excellent human-readable summary of the changes.

Of particular interest to embedded developers will be the new boot tracer facility, which allows to draw SVG graphs of the kernel initialization procedures execution time, in order to analyze the boot time and possibly reduce it. Of course, a lot of architecture-dependent improvements have also been made (for example OProfile support for ARMv7 CPUs but also new supported boards) and a lot of drivers have been merged or improved, as usual.

Bootlin has contributed a few patches that have been merged and released in 2.6.28. While being a small contribution compared to the 9.000+ patches added to the kernel between 2.6.27 and 2.6.28, they still slightly improve the kernel for embedded users. Part of the Linux-Tiny efforts, these patches allow to reduce the size of the kernel by disabling features that may not be necessary on embedded systems. More specifically, these patches allow :

From the existing Linux Tiny patch ideas, the only one left in the feature removal area are the multicast support removal and ethtool removal. They have already been submitted a few months ago, but got rejected by the network maintainers. I will work on them again to fix the issues and try to re-submit them later.

Finally, Jonathan Corbet has published an analysis of the 2.6.28 developement cycle in terms of contributors and changes. An interesting reading.