Linux init failures now easier to debug

If you are an embedded Linux developer too, you have probably been frustrated by the lack of information from the Linux kernel when it failed to start the init process when you’re building a new root filesystem. The only thing you get is No init found, and this could hide many different causes:

  • No init program candidate found at all
  • Some init program candidates exist but they can’t be executed, for multiple possible causes (missing execute permissions, failed to load shared libraries, executable compiled for an unknown architecture…)

The good news is that this source of frustration will be gone in Linux 3.13. Thanks to a Bootlin commit merged on Nov. 13, 2013, whenever an attempt to execute an init program candidate fails, there is a message in the console detailing the executable path and the error code. For example:

Starting init: /sbin/init exists but couldn't execute it (error -13)

When you get such a message, all you have to do is lookup the error code in include/uapi/asm-generic/errno-base.h or maybe in uapi/asm-generic/errno.h. In the above example, the -13 code meant permission denied, typically because of missing execution rights.

This had been annoying me for a long time, and I am glad that the Linux kernel community accepted my improvement!

By the way, many more improvements to the Linux kernel from Bootlin are currently getting merged in 3.13. See all our contributions to the Linux kernel.

Videos and slides of the Kernel Recipes 2013 conference

Kernel Recipes LogoAs we mentionned earlier on this blog, Bootlin participated to the second edition of the Kernel Recipes conference in Paris, a two-days conference dedicated to kernel topics.

The videos and slides of the talks in this conference have now been published, see https://kernel-recipes.org/en/2013/conferences/ for the complete list. There is a good number of interesting topics: discussion about kernel development environment by Willy Tarreau, status of Nftables and Netfilter in general by Eric Leblond, a talk explaning how to decipher kernel oopses, a talk about Crosstool-NG from Yann E. Morin, a discussion about Linux Security Modules, a talk about the status of Display support in the kernel by Laurent Pinchart, and several lightning talks.

The talks from Bootlin were:

Bootlin really enjoyed this conference, and is looking forward to participating again next year. Thanks a lot to the organizers!

Embedded Linux and kernel engineer job openings

Bootlin team

We’re getting busier than ever! Bootlin is looking for developers:

  • With experience developing embedded Linux systems
  • With experience developing device drivers for the Linux kernel, and porting Linux on new hardware. See our contributions to the mainline Linux kernel!
  • With technical writing skills and an interest for training

We need to fill at least 2 open positions in the next months, and more will follow in 2014.

Newly graduated engineers are welcome too, provided they already have experience in the above technical fields or with Free Software development.

This time, we are looking for people who will be able to join one of our offices in France (Toulouse or Avignon), to strengthen our engineering teams there.

  • Toulouse is a dynamic city with lots of high-tech and embedded systems companies in particular. Our office in Colomiers can easily be reached by train from downtown Toulouse if you wish to settle there. You would be working with Maxime Ripard and our CTO Thomas Petazzoni.
  • Our main office is settled in Orange in the heart of the Provence region, close to Avignon, a smaller but dynamic city too. It enjoys a sunny climate and the proximity of the Alps and the Mediterranean sea. Accommodation is very affordable and there are no traffic issues! You would be working with our founder Michael Opdenacker and of course remotely with the rest of the engineering team. In particular, we are interested in foreign engineers who could help us develop our services in their home countries.

If you are unable to relocate this time, don’t hesitate to contact us anyway. Depending on your profile and experience, we are still planning to open home based jobs in a few months or years from now.

If you are interested in these positions, here are nice opportunities to meet us in the next weeks:

See a full description and details about how to contact us.

Bootlin at the ARM Kernel Summit, the Embedded Linux Conference and the Buildroot Developers Meeting

Late october will be a busy moment for all the embedded Linux developers meeting in Edinburgh, UK. The Linux Foundation is organizing a number of conferences here, including the Embedded Linux Conference Europe (October 24-25) and LinuxCon Europe (October 21-23), and many co-located other events.

Bootlin will be present at several of these events:

  • First, three Bootlin engineers will be present at the ARM kernel summit on October 22nd and 23rd. The ARM kernel summit is an invitation-only conference, organized in relation with the Linux Kernel Summit. Gregory Clement, Maxime Ripard and Thomas Petazzoni, engineers at Bootlin have been invited due to their participation to the ARM support in the kernel, mainly on Allwinner SOCs for Maxime and on Marvell SOCs for Gregory and Thomas. Being present at this event is an excellent opportunity to be part of the discussion that shapes the future of ARM support in Linux, and strengthen our relations with other members of this growing community.
  • Then, the entire technical team of Bootlin will attend the Embedded Linux Conference, on October 24th and 25th. Several talks will also be given by Bootlin engineers:
    • On Thursday, 24th October at 11:40 AM, Thomas Petazzoni will give a talk titled Device Tree for dummies!, which will give an introduction to the Device Tree on ARM: what it is, how it is compiled, how it used by the kernel, how Device Tree bindings are defined, how drivers are affected by the Device Tree, etc.
    • At the same time in another room, Michael Opdenacker will lead a Bird of a Feather session dedicated to Small Businesses in the embedded Linux world. Exchanging experiences, networking with other companies working in the same field, etc.
    • Still on Thursday, at 3 PM, Gregory Clement will give a talk on the Linux kernel Common Clock Framework, which will be an updated version of the talk he gave at ELC earlier this year.
    • On Friday, 25th October at 9:30 AM, Thomas Petazzoni will be part of the keynote panel session dedicated to a discussion on Embedded Linux build systems together with Tim Bird (Sony Mobile), Ross Burton (Intel), and Karim Yaghmour (Opersys), the panel being moderated by Jeff Osier-Mixon (Intel).
  • On Saturday 26th and Sunday 27th October, the Buildroot community is organizing its traditional Developers Meeting, to which Thomas Petazzoni will participate. Some of the core Buildroot developers will join for two days of discussion and work to improve this embedded Linux build system.

As you can see, this will be a very interesting and busy week, and we’re all looking forward to meeting more embedded Linux developers and learning about the latest technologies in this field.

Bootlin the top #18 contributor to the 3.11 kernel

The 3.11 Linux kernel has now been released by Linus Torvalds, and as usual as thousands of patches coming from a large number of companies and contributors. For this release, Bootlin has contributed a total of 128 patches (yes, exactly 2^7), which makes Bootlin the 18th contributor in the list of companies contributing to the kernel, according to http://www.remword.com/kps_result/3.11_whole.html, before Broadcom and Cisco, and after ARM and Oracle. It is also the first time that six different engineers from Bootlin contribute code to the Linux kernel in a single release!

As usual, most of our contributions were centered around support for the Marvell Armada 370 and XP SOCs, the Allwinner SOCs and the Crystalfontz i.MX28 platforms:

  • Added support for the PCIe controllers of the Armada 370 and Armada XP platforms, and used it for the already supported Kirkwood platform. Supporting PCIe has been a very long process, which got started in December 2012, required long discussions with various kernel maintainers and multiple iterations of the patch series. Armada 370/XP was the first ARM platform to add Device Tree based PCIe support, and therefore this required many discussions to sort out the Device Tree bindings for PCIe controllers. This work was done by Thomas Petazzoni.
  • Enable an additional USB interface on the OpenBlocks AX3 platform, which is available as part of the mini-PCIe connector inside the device. This work was done by Thomas Petazzoni.
  • Cleaned up all the Kirkwood platform Device Tree files to assign the pin muxing configurations to the appropriate devices. This work was done by Thomas Petazzoni.
  • Made various cleanups and improvements in the Armada 370/XP platform code (in arch/arm/mach-mvebu) to make it possible to support different base address for the internal registers depending on the board being used. Many hardcoded physical addresses were removed, as well as the static virtual to physical mapping. This work was done by Thomas Petazzoni.
  • Cleaned up many ARM platforms to remove their unneeded ->init_irq() callback, and also the ->map_io() callback which we changed to default to calling debug_ll_io_init() when not provided. This work was done by Maxime Ripard.
  • Extended the ssd1307fb driver that we contributed a few releases ago to also support the SSD1306 device. The SSD1306 and SSD1307 are OLED screens controlled over I2C that are used on Crystalfontz i.MX28 platforms. We also optimized significantly the communication with the SSD130x devices. This work was done by Maxime Ripard.
  • Added an Ethernet driver for the Allwinner SOCs. The work was initially done by Stefan Roese, and our engineer Maxime Ripard did all the final cleanup, development of an MDIO driver, and integration with all the Device Tree files of the Allwinner platforms.
  • Added support for the Allwinner I2C controller, by re-using and extending the existing i2c-mv64xxx driver used on Marvell platforms, since the hardware block was very similar. The Allwinner Device Tree files were also updated to add the I2C controllers. This work was done by Maxime Ripard.
  • Added basic support for the Allwinner A10s SOC: pin muxing information and Device Tree information. This work was done by Maxime Ripard.
  • Added support for the Olimex A10s-Olinuxino-micro, a new hardware platform manufactured by Olimex that uses the Allwinner A10s SOC. This work was done by Maxime Ripard.
  • Implemented a “Device Bus” driver for the Marvell SOCs, that allows to configure the access to NOR flash and other devices connected to the memory bus. It has been used to enabled NOR support on the Armada XP DB development platform. This work was done by Ezequiel Garcia.
  • Fixed a few bugs in the IIO subsystem, and a build failure on AT91 platform when CONFIG_PHYLIB was not enabled. This work was done by Alexandre Belloni.
  • Fixed the ARM low-level code that handles compatibility with ATAG bootloaders, to properly convert 32 bits memory sizes passed by the bootloader into 64 bits cells of the Device Tree, when LPAE is used. This work was done by Gregory Clement.
  • Michael Opdenacker made a few improvements and fixes to the documentation.

For the upcoming 3.12, we already have 131 patches lined up, and a few more will probably show up after this blog post is written. Over the last release cycles, Bootlin has become a regular contributor to ARM support in the Linux kernel, and we’re looking forward to doing more contributions in the future.

The details of our 3.11 contributions is:

Starting Linux directly from AT91bootstrap3

Here is an update for our previous article on booting linux directly from AT91bootstrap. On newer ATMEL platforms, you will have to use AT91bootstrap 3. It now has a convenient way to be configured to boot directly to Linux.

You can check it out from github:

git clone git://github.com/linux4sam/at91bootstrap.git

That version of AT91bootstrap is using the same configuration mechanism as the Linux kernel. You will find default configurations, named in the form:
<board_name><storage>_<boot_strategy>_defconfig

  • board_name can be: at91sam9260ek, at91sam9261ek, at91sam9263ek, at91sam9g10ek, at91sam9g20ek, at91sam9m10g45ek, at91sam9n12ek, at91sam9rlek, at91sam9x5ek, at91sam9xeek or at91sama5d3xek
  • storage can be:
    • df for DataFlash
    • nf for NAND flash
    • sd for SD card
  • our main interest will be in boot_strategy which can be:
    • uboot: start u-boot or any other bootloader
    • linux: boot Linux directly, passing a kernel command line
    • linux_dt: boot Linux directly, using a Device Tree
    • android: boot Linux directly, in an Android configuration

Let’s take for example the latest evaluation boards from ATMEL, the SAMA5D3x-EK. If you are booting from NAND flash:

make at91sama5d3xeknf_linux_dt_defconfig
make

You’ll end up with a file named at91sama5d3xek-nandflashboot-linux-dt-3.5.4.bin in the binaries/ folder. This is your first stage bootloader. It has the same storage layout as used in the u-boot strategy so you can flash it and it will work.

As a last note, I’ll had that less is not always faster. On our benchmarks, booting the SAMA5D31-EK using AT91bootstrap, then Barebox was faster than just using AT91bootstrap. The main reason is that barebox is actually enabling the caches and decompresses the kernel(see below, the kernel is also enaling the caches before decompressing itself) before booting.

Bootlin contributions to the 3.10 kernel

The 3.10 Linux kernel has been released a few days ago. According to LWN, with almost 13.500 non-merge commits, the 3.10 has been the busiest ever, and also the fastest. Bootlin engineers again contributed to this release, with 99 patches integrated, making Bootlin the 28th most active company contributing, right between ST-Ericsson (103 patches) and ARM (97 patches). See http://www.remword.com/kps_result/3.10_whole.html for the complete statistics.

This time, Bootlin contributions include:

  • LPAE support for the Marvell Armada XP SoC, done by Grégory Clement.
  • Fix for errata 4742 of the PJ4B CPU core (used in Armada 370/XP), which prevented booting Armada 370 platforms after ARM optimized some TLB operations. Done by Grégory Clement.
  • Support for NOR flash on Marvell Armada 370 and Armada XP SoC, done by Ezequiel Garcia
  • Addition of a mvebu-mbus driver to handle the address decoding mechanism and configurable memory windows of Marvell SoC. The mach-kirkwood, mach-orion5x, mach-dove, mach-mv78xx0 and mach-mvebu Marvell platforms are all converted to use it. Developping this driver was a requirement to enable PCIe in a Device Tree compatible way on these platforms. Done by Thomas Petazzoni.
  • Addition of Device Tree information for the PCIe controllers of the Armada 370 and Armada XP, but unfortunately not the PCIe driver itself (which will arrive in 3.11). Done by Thomas Petazzoni.
  • Support for the thermal sensor on Marvell Armada 370 and Armada XP SoC, done by Ezequiel Garcia
  • A lot of reorganization of the Device Tree compatible strings for the Allwinner ARM SoC support, to prepare for the addition of additional SoCs in the future. Done by Maxime Ripard.
  • Improvements to the Allwinner pinctrl driver, with support for the A10 and A13 SoC. Done by Maxime Ripard.
  • Enabling of the I2C GPIO expander of the Armada 370 based Mirabox platform. Done by Grégory Clement.
  • A few updates to the support for the i.MX28 Crystalfontz boards: touchscreen and one-wire support on CFA10049. Done by Alexandre Belloni.
  • Various cleanups and improvements to the OMAP GMPC driver, done by Ezequiel Garcia.
  • Various cleanups and improvements to the Marvell Armada 370/XP IRQ controller driver, done by Thomas Petazzoni.

In detail, the contributions are:

Bootlin contributions to the 3.9 kernel

A few months ago, we published a blog post showing our contributions to the 3.8 Linux kernel. With 128 commits merged in 3.8, Bootlin was ranked as the 17th company in terms of kernel contributions.

The 3.9 kernel has been released a few weeks ago, with again a significant number of contributions from Bootlin. According to these statistics, Bootlin contributed 92 patches during the 3.9 cycle, making the company the 26th most important contributor to the Linux kernel for this release, and this time, five engineers from Bootlin contributed patches.

Among the contributions that we made:

  • Added a basic infrastructure for irqchip drivers in the drivers/irqchip directory. This directory is now used to store the drivers for the IRQ controllers of various processors.
  • Made a number of improvements to the Marvell SDIO driver, including the addition of a Device Tree binding for it, and enabled its usage on Marvell Armada 370 and Armada XP platforms, as well as converting the Marvell Kirkwood platforms to use Device Tree probing instead of legacy probing for their SDIO interface.
  • Contributed a number of improvements to support Crystalfontz i.MX28 based modules, including the Device Tree for the CFA10037 expansion board, various improvements for the CFA10049 expansion boards, and a driver for the Himax HX8357B LCD controller.
  • A large number of improvements to the support of the Allwinner ARM SoCs, most notably a pinctrl driver for those SoCs, which allows to configure the muxing of I/O pins, and a gpio driver, to use the pins as general-purpose I/Os. We also contributed the support for the Miniand Hackberry platform, based on an Allwinner SoC. This work is all done by our engineer Maxime Ripard, who is the maintainer of the Allwinner SoC support in the Linux kernel.
  • Improvements to the PCA953x driver (for I2C GPIO expanders) in order to support the PCA9505 chip, that has 40 GPIOs. This required quite some work, as the PCA953x was originally limited to chips having at most 32 GPIOs. This improvement was done in order to support the GPIO expander box provided by Globalscale for the Armada 370-based Mirabox platform.
  • We added support for the Real Time Clock on Armada 370 and Armada XP based platforms, added support for local timers on Armada XP, added support for the new Armada XP GP evaluation board.
  • We enabled support for the SPI controllers and the USB controllers on Armada 370 and Armada XP based platforms.

Our high rate of contributions is going to continue, as we already have 95 patches merged for the upcoming 3.10 kernel and have already submitted a number of patches for the 3.11 kernel.

Here are details about our contributions to the 3.9 kernel:

Linux kernel 3.8 released, Bootlin top #17 contributor

Thomas Petazzoni and Grégory Clement, Bootlin kernel engineers
Thomas Petazzoni (front) and Grégory Clement (back) at the Embedded Linux Conference 2013 in San Francisco, discussing ARM Linux kernel issues.
Early last week, version 3.8 of the Linux kernel has been released by Linus Torvalds. The KernelNewbies web site, has, as usual, a great summary of what’s new in this release, together with lots of links to the relevant LWN articles. With 12394 commits, 3.8 has been the busiest ever kernel release cycle, the previous record being held by 2.6.25 with 12243 commits.

Despite this huge activity, Bootlin has been the 17th most active employer during the 3.8 cycle, with 128 commits merged into the mainline Linux kernel, representing a bit more than 1% of the total number of commits. See the statistics by employer at http://www.remword.com/kps_result/3.8_whole.html and in the traditional LWN article. This puts Bootlin before Nvidia, Qualcomm, ARM or Oracle in number of commits, and just a few commits behind Freescale. See the Git repository for the list of our contributions.

In detail, Bootlin contributions for 3.8 have been:

  • A large number of contributions related to the support of the Marvell Armada 370 and Armada XP SoCs, done by Grégory Clement and Thomas Petazzoni. Contributions included: a new network driver for the Armada 370 and Armada XP, support for the Armada XP-based OpenBlocks AX3 platform, support for the Armada 370-based Globalscale Mirabox platform, a big number of improvements and Device Tree support for the Marvell XOR engine driver, beginning of Device Tree support for the older Marvell Orion5x SoC family, support for the L2 cache found in Armada 370/XP, clock drivers for Armada 370/XP, SMP support for Armada XP, enabling of SATA on Armada 370/XP platforms.
  • The contribution of the initial support for a new SoC family in the mainline Linux kernel: the Allwinner A10 and Allwinner A13 ARM SoCs. This support has been contributed by Maxime Ripard, who has become the maintainer for this new ARM sub-architecture.
  • A driver for the I2C-based SSD1304 OLED display, a nice 128×32 pixels monochrome OLED display, contributed by Maxime Ripard.
  • A number of improvements in the support for the Crystalfontz i.MX28-based platforms, the CFA10036 and its expansion board the CFA10049. These contributions have also been made by Maxime Ripard.

Through these contributions, Bootlin have gained a good expertise in support for ARM SoCs and boards inside the Linux kernel. If you are interested in having us help you bring the support of your ARM board or ARM SoC into the mainline Linux kernel, do not hesitate to contact us, you will be directly answered by our engineers doing Linux kernel development!

Bootlin customer project on Kickstarter!

For about 6 months, we’ve been working with Crystalfontz America on an imx28-based board, targeted at the hackers and DIYers. We’ve been working on the BSP, adding support to Linux and in Buildroot for this board. Support in the mainline Linux kernel is also in pretty good shape, and we continue to post patches to improve it.

The CFA-10036 is actually a computer-on-module with a small OLED display, and comes with two (for now) breakout boards, the CFA-10037, which adds USB and Ethernet connectivity, and an awful lot of exposed GPIOs, and the soon-to-be announced CFA-10049, which is more targeted to industrial or robotic uses, with additional ADCs, fan controller, 1-wire, LCD, rotary encoder, and so on. See more details.

The project is getting close to completion, since Crystalfontz started its funding campaign on Kickstarter.

For those who are not familiar with Kickstarter, it’s a way for creators to get funding and sense customer interest in their projects. If you find the device interesting you can either make a small pledge to show that you like the project, or make a bigger one and will receive board(s) and accessories corresponding to how much you pledged. If the project doesn’t meet its funding goals, you won’t be charged at all. I advise you to read the Kickstarter FAQ to understand Kickstarter better.