Embedded Linux Conference day 1

The first day of the Embedded Linux Conference started on Wednesday here at Redwood City, California.

The day started with the usual Kernel Report from Jonathan Corbet. It was, as usual with Corbet’s talk, a very interesting summary of what happened in the kernel through the last year, with highlights of the major new features per release, thoughts about issues like the kernel.org security problem and subsequent outage, etc.

The Kernel Report, Jonathan Corbet
The Kernel Report, Jonathan Corbet

After this talk, Grégory went to the Saving the Power Consumption of the Unused Memory talk, given by Loïc Pallardy, who works for ST Ericson in France. The purpose of the talk was to detail the kernel modifications they made to support the fact of powering down portions of the memory that are unused. In fact, DDR memories these days are capable of powering off some their areas, which allows to save power. Of course, when an area of the memory is powered off, its contents are lost, so the kernel needs to ensure that nothing valuable remained on this area of memory. Their kernel modifications allow to describe how the memory is organised (which address ranges are available and can be powered down independently) and introduce some kernel memory allocator changes to reference count those banks of memory. Of course, the next problem is that physical memory is usually highly fragmented, so they detailed how they re-used some of the existing kernel mechanisms to group unmovable pages on one side and movable pages on the other side and that allow to defragment the movable pages. This topic has been worked on since quite a long time in the kernel, as can be found in this LWN article from 2006.

Saving the Power Consumption of the Unused Memory, Loïc Pallardy
Saving the Power Consumption of the Unused Memory, Loïc Pallardy

On my side, I attended the What Android and Embedded Linux Can Learn From Each Other talk. The speaker detailed many of the Android kernel additions and how they could, theoretically, be re-used in non-Android embedded Linux systems. Things like re-using the Binder inter-process communication mechanism, or simple things like the RAM-based Logger mechanism. Unfortunately, none of the speaker’s suggestions were backed by any sort of real experimentation, so those suggestions were mostly speculations. For example, he suggested the possibility of re-using the Android graphics stack on a non-Android system, but most likely this is a very difficult task to achieve and not necessarily worth the effort. At the end of the talk, the speaker suggested that the embedded Linux community and the Android community should talk more to each other, but looking at how Google is driving Android development, it is difficult to see this happening in the near future.

Then, the talk from Hisao Munakata about Close Encounters of the Upstream Resource was an interesting and good summary of the tensions that exist within embedded companies between the product teams (who have deadlines and need the product to work, and don’t want to worry about upstreaming things) and the community teams (who are in relation with the community and try to upstream modifications). He had really nice slides to show the multiple issues that a company faces when it produces major modifications to open-source components such as the Linux kernel, without any effort to upstream them. But he also said that things are improving, and that with Android using fairly recent kernel versions, the embedded Linux system makers are now much closer to mainline versions, which helps in getting changes merged in the official Linux kernel. He advocated that embedded Linux developers should be proficient with git, because it allows to easily track the modifications, find out whether bugs have been fixed in later versions of the Linux kernel, etc. He also quickly presented LTSI, an initiative that offers long-term support around the Linux kernel. He presented it as the way of solving the fragmentation between the vendor BSPs kernel versions, the Android versions, and all other kernel versions that are floating around. However, how those versions will get merged into the official Linux kernel was not really clear.

In the afternoon, Grégory went to the talk Comparing Power Saving Techniques For Multicore ARM Platforms, presented by Vincent Guittot was an other talk presented by a French guy from ST Ericsson. As the one Grégory saw in the morning about power management of memories, this one was also very instructive, well documented and the speaker seemed to really know his topic. He worked the right way on Linux: only very minimal changes inside the kernel, tried to reuse the existing components, provided a git tree available and proposed some improvements on the mailing lists: good job!

Grégory also attended the traditional talk from Tim Bird entitled Status of Embedded Linux. Very pleasant talk (as usual with Tim Bird). It was a very good overview of the state of embedded Linux. If you want to start working on embedded Linux this talk is a must see. Moreover Tim mentioned the valuable work done by Bootlin by recording and sharing the conferences for many years!

The Status of Embedded Linux, Tim Bird
The Status of Embedded Linux, Tim Bird

Later in the day, I attended the talk Passing Time With SPI Framebuffer Driver given by Matt Porter, who now works for Texas Instruments. His talk was feedback from real-life experience developing a driver for a SPI framebuffer controller. Initially, the problem was that a customer had started developing a driver, but that driver violated all the Linux development rules: no usage of the GPIO APIs, no usage of the SPI infrastructure, no usage of the device model, everything was done through a basic character driver directly manipulating the hardware registers. This is something that we also see quite sometimes at Bootlin in the kernel code of some customers: this happens when the code has been written by developers who have only started reading the Linux Devices Driver book, but didn’t go far enough in the Linux code to understand the device model and the principle of code re-usability. So clearly, Matt’s experience resonated with our own experience. So, Matt went on to describe how the driver worked, modifications needed at the board configuration level, the driver itself, its integration in the device model. He also clearly detailed how a SPI framebuffer can work. On a normal framebuffer integrated into the SoC, the framebuffer memory is directly mapped into the application address space so that the application can directly draw pixels on the screen. However, when the framebuffer controller is over SPI, it is clearly not possible to map the framebuffer memory into the application address space. But fortunately, the kernel has a dedicated mechanism for such case: FB deferred I/O. What gets mapped into the application address space is normal kernel memory, but the kernel detects thanks to page faults when a portion of this memory has been changed, and calls the framebuffer driver so that the driver has an opportunity to push these changes over SPI to the framebuffer controller. Of course, this mechanism run at a configurable frequency. The device that was used by Matt Porter was a 1.8 screen available from Ada Fruit, this might also been a good device to use in our future kernel courses, to let participants exercise with driver development.

At the end of the day, I attended the Experiences With Device Tree Support Development For ARM-Based SOC’s by Thomas P. Abraham, from Samsung Electronics, but also from Linaro. It was clearly an excellent presentation about the device tree and how it works. It showed, with lots of code examples, how to compile the device tree source into a device tree blob, how to configure and use U-Boot to get this device tree blob loaded and passed to the kernel, how the board files in the kernel are changed to use the device tree, how device drivers are modified, how the platform data mechanism is changed with the device tree, and more. Definitely a must-see for anyone doing ARM development these days.

My colleague Maxime went to the talk from Paul McKenney about Making RCU Safe For Battery-Powered Devices. Maxime reported that it was an excellent introduction to RCU: Paul introduced very progressively the various issues, so it was possible even for an RCU-newbie to follow that talk. Definitely a presentation I will watch thanks to the video recording!

In the evening, there was the traditional social event of the conference. It took place at the Hiller Aviation Museum, they have lots of strange aircrafts or helicopters, such as a piece of the supersonic Boeing prototype plane, or other bizarre flying devices such as this flying platform.

Android Builders Summit: day 2, and Yocto Developer Day

It’s now Thursday morning here in Redwood City, California, and I didn’t had the time yesterday morning to do a write-up about our second day at the Android Builders Summit. Hopefully the following write-up will give our readers some details about what happened during this day.

This second of Android Builders Summit was co-located with the Yocto Developer Day, and as my colleagues Maxime Ripard and Grégory Clément were attending the two tracks of Android Builders Summit, I decided that I would attend the Yocto Developer Day.

Yocto Developer Day

Yocto is an umbrella project that consists in creating an embedded Linux build system, called Poky and some associated development tools. Poky takes its roots into the OpenEmbedded community: it re-uses the bitbake recipe processor, and a set of recipes for building packages that are now shared between Yocto and OpenEmbedded through the openembedded-core repository. At Bootlin, we are strong contributors to the Buildroot build system, and we use it for many of our customer projects. However, being pushed very strongly by Intel and the Linux Foundation, Yocto is gaining traction, and the fact that Yocto provides stable releases every 6 months now makes it a lot more usable than OpenEmbedded, which had to be checked out through Git, leaving the user with the uncertainty on whether the version (s)he got would actually work or not. And moreover, Buildroot and Yocto are not really competing projects: Buildroot is a simple root filesystem image generator, while Yocto is more a cross-distribution generator, they target projects of different sizes.

I started attending the Yocto Developer Day with a general presentation talk about what Yocto is and why it is necessary. Nothing really new in this talk for someone who already uses embedded Linux build systems and understands the need for such tools. However, the thing that always surprises me is that the Yocto project claims everywhere to solve the fragmentation problem in the embedded Linux build system space (there are too many tools in this area) by creating the tool, and that they envision that in 5 years, everybody will link embedded Linux build with the Yocto project. It’s quite funny because at the moment, they have just created yet another build system 🙂 But it’s true that the project is gaining traction and seems to attract the SoC vendors, which is a good thing because having a standard build system is so much better than having crappy vendor-specific build systems.

Yocto Developer Day: Yocto introduction by Saul Wold from Intel
Yocto Developer Day: Yocto introduction by Saul Wold from Intel

The second talk, by Saul Wold, from Intel, went more into the details on how to use Yocto: what the different components are, how recipes are written, how configuration is defined, what tasks, images, recipes, etc are. I would have liked if the talk went a bit further into the details, but it gave a very good introduction to the Poky build system.

In the afternoon, I attended a hands-on session for new users to Yocto. The room setup was very impressive: about forty high-end PCs provided, each having a development board next to it. The first part of the hands-on session consisted in using Yocto to produce a basic filesystem image which we booted into Qemu. In order to solve the very long first build problem that all OpenEmbedded and Yocto users face, they had pre-built a number of packages and stored them into a shared state directory. Interestingly, the size of the Yocto output directory was about 30 GB, just to build an embedded Linux system with BusyBox and a few minor things. Once this was done, we went ahead in creating our own layer, in order to define our own image and its contents it terms of packages. We used it to add a graphical splashscreen, and I also extended it to include Dropbear into the build. The whole thing went quite well. One thing that worries me is that bitbake and the build process really looks like a black box, and it seems hard to understand what’s going on behind the scenes. With Buildroot, I am used to a very simple build system with which it is very easy to fully understand what’s going on. Here, even the people that give speeches about Yocto or deliver a bit of training, seem to not fully understand what’s going on. This impression is also validated by the complexity of the output directory (where all the build results are). But maybe it’s just a matter of spending some time using it and reading some code, but the fact that people that have been developing/using Yocto for a while still do not really understand its internals is a bit surprising. Or maybe it’s just a wrong impression on my side.

Yocto Developer Day: hands-on session starting
Yocto Developer Day: hands-on session starting

The next part of the hands-on was around the Eclipse integration of Yocto. First with ADT (Application Development Toolkit), which integrates the cross-development thing into Eclipse. Thanks to an agent running into the target, Eclipse is able to push the application binary to the target and start gdbserver on it, and therefore transparently start a debugging session for the user. I am not a big fan of Eclipse (I have been an Emacs user for a huge number of years), but it’s true that for people used to Integrated Development Environments, this ADT thing provides a quite nice experience. Then, we went ahead in trying to use HOB, which integrates into Eclipse the possibility of selecting which packages should be built and integrated into the image. Unfortunately, it seems it didn’t work for anybody (even though we were selecting the package in the GUI, it didn’t appear in the final filesystem image), but that wasn’t a big problem since I don’t really see the point in a tool such as HOB: editing configuration files is something that shouldn’t scare any embedded Linux developer.

Regardless of the contents of the hands-on, I was quite interested by how it was conducted. Instead of having some written lab instructions, and having everyone following, alone, those lab instructions, the instructor was simply demoing the various steps to be done on the video-projector screen, which we simply had to replicate. It makes the session quite interactive, with of course the drawback that everyone needs to progress at the same pace.

All in all, this Yocto Developer Day was interesting, and I hope to find some time soon to experiment further with Yocto.

Android Builders Summit

My colleagues attended multiple talks about Android during this second day of the conference. In the morning, they attended Headless Android, Towards a Standard Audio HAL for Android, Android on eMMC: Optimizing for Performance.

Android Builders Summit: Real-time Android
Android Builders Summit: Real-time Android

In the afternoon, Grégory attended the Real-Time Android talk, which he said, was interesting. It showed that it was possible to integrate the PREEMPT_RT patches together with the Android kernel modifications, and provide a system having real-time capabilities for native (C/C++) applications and still the nice aspect of the Android user interface. During the same slot, Maxime attended the Android Services Black Magic, given by Aleksandar (Saša) Gargenta from Marakana. As usual with the Gargenta brothers, the talk was highly interesting and gave a lot of detailed information about Android services.

Some other thoughts…

At the organization level, the conference organizers should make it clear in the conference program and flyer the location where the slides will be posted. At almost every talk there is someone that asked if and where the slides will be posted, and the speakers are sometimes a bit uncomfortable because there is no clearly identified place to post the slides. In the past years, it was made clear that the slides would be posted on the elinux.org wiki, but this year, things are very unclear. Moreover, it’s even more surprising since speakers are asked to post their slides into their Linux Foundation website account, but those slides are not being made visible. Maybe a good suggestion for the Linux Foundation would be to improve how slides are handled and posted online.

Another thought about the Android Builders Summit is the surprising absence of Google, the developer and maker of Android. Google sponsors the Embedded Linux Conference which takes place right after the Android Builders Summit, but they do not sponsor the Android Builders Summit. There is also no talk from Google developers, and I haven’t seen any Google person in the attendees. It’s even more surprising when we know that the conference takes place in a location about 18 minutes away by car from Google headquarters in Mountain View. Maybe Google doesn’t want to see Android being used in application areas other than phones and tablets?

First day at the Android Builders Summit

Yesterday was the first day at the Android Builders Summit 2012, here in Redwood City, near San Francisco, California. My colleagues Grégory Clément and Maxime Ripard as well as myself are fortunate to attend this conference, and the contents of the first day were really interesting.

Amongst others:

  • A talk from Karim Yaghmour, well-known for having worked on the original version of the Linux Trace Toolkit, on the Adeos patch, as well as for its activity around Android. He delivered a 30 minutes talk about Leveraging Linux’s history in Android, which covered the differences in architecture between a standard embedded Linux system and Android, as well as how to nicely integrate BusyBox or tools like the Linux Trace Toolkit into Android. The presentation was really impressive: in just 30 minutes, Karim covered a huge number of slides, and made several live demonstrations. It is also worth noting that Karim, following the direction that Bootlin has drawn 7 years ago, has decided to release his Android training materials under a Creative Commons BY-SA license.
  • A panel with multiple kernel developers and people involved in Android on how to integrate the specific Android kernel patches into the mainline kernel. Not many new things learned here: the issue with the Android patches is that they add a lot of new userspace-to-kernel APIs, and such code is much much harder to get in mainline than conventional driver or platform code, since such APIs need to be maintained forever. Interestingly, Zach Pfeffer from Linaro pointed out the fact that the major problems with Android integration these days are not due to the kernel patches, but rather to the horrible binary blobs and related drivers that are needed for 3D acceleration ARM SoCs (Systems On a Chip).
    Panel « Android and the Linux Kernel Mainline: Where Are We? »
    Panel « Android and the Linux Kernel Mainline: Where Are We? »
  • A talk from Marko Gargenta on how to customize Android. He explained how to expose a specific Linux kernel driver functionality to Android applications, through a native C library, the JNI mechanism and an Android service, with complete details in terms of source code and build system integration. This presentation, just like last year’s presentation from Marko, was absolutely excellent. A lot of content, very dynamic presentation, a lot of things learned.
  • A talk on how ADB (Android Debugger) works. The contents were really good as well here, with lots of details about the ADB architecture, some tips and tricks about its usage, and more. Unfortunately, the speaker was really not familiar with English, and most of its presentation was spent reading the slides. This is a bit unfortunate because the technical contents was really, really excellent. The slides are available at http://www.slideshare.net/tetsu.koba/adbandroid-debug-bridge-how-it-works.
  • Using Android in safety-critical medical devices. This talk was not about technical issues, but rather about the reason for using Android in medical devices (get those devices connected together and collect some data to learn more about medical practices, their efficiency and cost) and also the legal requirements to get such devices validated by the Food and Drugs Administration in the US. A lot of useful arguments on how to convince managers that Android and Linux in general are usable in safety-critical medical devices.
  • A talk about Over-The-Air updates in Android, which I didn’t attend, but my colleague Maxime Ripard and other attendees gave an excellent feedback about it. It detailed an advanced system for safely upgrading an Android system, using binary diffs and other techniques.
    Customizing Android by Marko Gargenta, Marakana
    Customizing Android by Marko Gargenta, Marakana
  • The talk about Integrating Projects Using Their Own Build System Into the Android Build System had a really promising title and abstract, but unfortunately, the contents were disappointing. The speaker took 25 minutes just to explain how to build BusyBox (outside of any Android context) and then another 20 minutes to explain how to integrate it in the Android build system, on unreadable slides.
  • The talk about Android Device Porting Walkthrough was really great. Benjamin Zores exhausted its time slot with a 1h15 talk instead of the 50 minutes slot allocated, but fortunately, it was the last talk of the day in this session. During this talk, Benjamin gave a huge amount of information and many details about various issues encountered in the process of adapting Android for an Alcatel business VoIP phone (the ones you see in business desks). Issues like filesystem layout, input subsystem configuration, touchscreen configuration, graphics and much much more were covered. Be sure to check out Benjamin slides at http://www.slideshare.net/gxben/android-device-porting-walkthrough.
  • Finally, the day ended with a lightning talk session moderated by Karim Yaghmour. Lightning talks are really nice, because in less than 5 minutes, you quickly hear about a project or an idea. When the speaker is not good or the topic uninteresting, you know that after 5 minutes, you’ll hear someone else speaking about a different topic. The lightning talk on the integration of GStreamer in Android was really interesting, as was the lightning talk from Karim about its CyborgStack initiative, which creates an upstream Android source to integrate all the Android modifications that will never be mainlined by Google. See Karim slides at http://www.cyborgstack.org/sites/default/files/cyborgstack-120213.pdf for details.

And now, it’s time for breakfast, before the conferences of the second day of this Android Builders Summit.

HOWTO – Mailing lists with Mailman on Ubuntu 10.04

GNU mailman logoBootlin is not in the system administration business (we offer free and open-source solutions for embedded systems), but we do our best to share whatever experience we acquire, and whatever code we produce.

We configured a KVM virtual machine to run our mailing lists, and we used Ubuntu 10.04, the long term support (LTS) version. Here are instructions based on this experience. You could also use the same instructions to install Ubuntu 10.04 on a tiny, low power ARM board. And if you wish to use a more recent version of Ubuntu, I expect the steps to be very similar.

Here we assume that your domain is example.com and that you install your mailing list software on lists.example.com. It is indeed a good idea to install your mailing list software on a different server. This way, you won’t mess up with your main web and mail servers. Not having CGI scripts running on it will also keep your main mail server more secure.

Install packages

Install the Apache web server. It will allow administrators to configure and manage the lists, and users to subscribe, unsubscribe and tune their subscription settings.

sudo apt-get install apache2

For e-mail delivery, I chose the Postfix MTA (Mail Transport Agent). Exim would have been a good solution too, but I am more familiar with Postfix, which is already in use on our main mail server.

sudo apt-get install postfix

Choose the Internet site option.

Now install the mailman package:

sudo apt-get install mailman

Select the languages that you want your mailing list interfaces to support. I chose English and French for the moment. To add more languages later, run:

sudo dpkg-reconfigure mailman

Configure the Apache web server

First copy the sample Apache configuration file provided by the mailman package:

cp /etc/mailman/apache.conf /etc/apache2/sites-available/mailman

Now, in /etc/apache2/sites-available/mailman, enable short URLs by enabling:

ScriptAlias /mailman/ /usr/lib/cgi-bin/mailman/

Also modify /etc/mailman/mm_cfg.py:

DEFAULT_URL_PATTERN = 'http://%s/mailman/'

The last step is to enable your mailman site in Apache:

sudo a2ensite mailman
sudo /etc/init.d/apache2 restart

You should now have a new symbolic link in /etc/apache2/sites-enabled/.

Check that the Mailman website works by opening your mailing lists home page: http://lists.example.com/mailman/listinfo. It should look like https://lists.bootlin.com/mailman/listinfo.

Configure postfix

Enable the following line in /etc/mailman/mm_cfg.py:

MTA='Postfix'

Once the MTA is configured, generate Mailman specific aliases for Postfix:

sudo /usr/lib/mailman/bin/genaliases

Now, you need to configure Postfix through its main.cf file. A convenient way to do this is to run the below commands:

sudo postconf -e 'relay_domains = lists.example.com'
sudo postconf -e 'transport_maps = hash:/etc/postfix/transport'
sudo postconf -e 'mailman_destination_recipient_limit = 1'
sudo postconf -e 'alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases'

Also add the following line to /etc/postfix/transport:

lists.bootlin.com      mailman:

and run:

sudo postmap -v /etc/postfix/transport

You won’t have any mail delivery if you forget. I struggled for a few hours before I realized I forgot this setting.

Now, set correct file ownership:

sudo chown root:list /var/lib/mailman/data/aliases
sudo chown root:list /etc/aliases

For logging and debugging e-mail delivery, I recommend to install the sysklogd package. Without it, you won’t have any mail.info, mail.warn and mail.err files in /var/mail/.

To install this package, enable the universe repository if needed (uncomment the lines with universe in the /etc/apt/sources.list file), and run:

sudo apt-get update
sudo apt-get install sysklogd

To avoid having two mail.info and mail.log files with identical contents, edit /etc/syslog.conf and remove the below line:

mail.*                          -/var/log/mail.log

Also create a /etc/logrotate.d/mail file to rotate logs, as in the below example:

/var/log/mail.* {
        daily
        size 10M
        rotate 4
        compress
        missingok
        notifempty
        create 640 root adm
}

Last but not least, restart Postfix:

/etc/init.d/postfix restart

At this point, a good idea is to check that mail delivery works:

sudo apt-get install bsd-mailx
mailx alice@example.com
Subject: test
test
.
Cc: 

Note: that’s the line containing only a dot character that allows to terminate the message.

If the receipient doesn’t receive this message, there is an issue in the way your mail server is configured. This could be because the firewall doesn’t allow connections to outside machines through tcp port 25. Anyway, look at the logs in /var/log/mail.* to get a clue. There is no point going on in this howto until you get this fixed.

Creating the mailman site list

Mailman needs a so-called “site list”, which is the list from which password reminders and such are sent out from. The default name for this list list mailman, though you can change this through the MAILMAN_SITE_LIST setting in /etc/mailman/mm_cfg.py.

To create this list, run:

sudo newlist mailman

You will have to answer a few questions like:

Enter the email of the person running the list: postmaster@example.com
Initial newsletter password: xxx
Hit enter to notify newsletter owner...

Choose the password carefully, as crackers will be able to highjack your mailing list if it is too easy to guess.

The next required step is to add the list aliases to /etc/aliases:

# mailman mailing list
mailman:              "|/var/lib/mailman/mail/mailman post mailman"
mailman-admin:        "|/var/lib/mailman/mail/mailman admin mailman"
mailman-bounces:      "|/var/lib/mailman/mail/mailman bounces mailman"
mailman-confirm:      "|/var/lib/mailman/mail/mailman confirm mailman"
mailman-join:         "|/var/lib/mailman/mail/mailman join mailman"
mailman-leave:        "|/var/lib/mailman/mail/mailman leave mailman"
mailman-owner:        "|/var/lib/mailman/mail/mailman owner mailman"
mailman-request:      "|/var/lib/mailman/mail/mailman request mailman"
mailman-subscribe:    "|/var/lib/mailman/mail/mailman subscribe mailman"
mailman-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe mailman"

In addition to delivering e-mail to the mailing lists, these aliases also allow to subscribe and unsubscribe by writing to special e-mail addresses.

You also need to run the newaliases command, without which there is no e-mail delivery:

sudo newaliases
sudo /etc/init.d/postfix restart
sudo /etc/init.d/mailman restart

Look at the /var/log/mailman/error file for potential issues.

Create regular mailing lists

Regular mailing lists are created in the same way as above: Assuming you want to create a newsletter mailing list. You will need run:

sudo newlist newsletter

Add your new mailing list to /etc/aliases:

# newletter mailing list
newsletter:              "|/var/lib/mailman/mail/mailman post newsletter"
newsletter-admin:        "|/var/lib/mailman/mail/mailman admin newsletter"
newsletter-bounces:      "|/var/lib/mailman/mail/mailman bounces newsletter"
newsletter-confirm:      "|/var/lib/mailman/mail/mailman confirm newsletter"
newsletter-join:         "|/var/lib/mailman/mail/mailman join newsletter"
newsletter-leave:        "|/var/lib/mailman/mail/mailman leave newsletter"
newsletter-owner:        "|/var/lib/mailman/mail/mailman owner newsletter"
newsletter-request:      "|/var/lib/mailman/mail/mailman request newsletter"
newsletter-subscribe:    "|/var/lib/mailman/mail/mailman subscribe newsletter"
newsletter-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe newsletter"

Then, run the usual commands:

sudo newaliases
sudo /etc/init.d/postfix restart
sudo /etc/init.d/mailman restart

Configuring your lists

The easiest way to configure your lists and add members is to open the http://lists.example.com.com/mailman/listinfo URL with a browser.

A few things are also possible from the command line. For example, you can add a member as follows:

echo "alice@example.com" > /tmp/foo
sudo add_members -r /tmp/foo newsletter

Enabling archives

Default file permissions are not completely ready to support mailing list archives:

sudo chown -R root:list /var/lib/mailman/archives
sudo chmod o+rX /var/lib/mailman/archives/private

The second line allows the webserver to access the archives. Note that these settings were forgotten in the official Ubuntu documentation.

Settings for newsletter mailing lists

At Bootlin, we also use Mailman to deliver our newsletters. Mailman provides recipients with an easy mechanism to subscribe by themselves and unsubscribe whenever they want.

To make it even easier to unsubscribe from a newsletter, Mailman can add a special footer to each recipient, with a custom URL that allows to unsubscribe without having to remember one’s password.

To enable this feature, you have to enable the following line in /etc/mailman/mm_cfg.py:

# Extra options
# Allow to personalize each message
# (useful to provide a password-less unsubscribe link)
OWNERS_CAN_ENABLE_PERSONALIZATION = 1

Restart Mailman (/etc/init.d/mailman restart) and go to the administrative interface for your list. Under Non digest options, you will then be able to set the Should Mailman personalize each non-digest delivery? option to Full Personalization.

Then, you can set a custom footer for each recipient in the Footer added to mail sent to regular list members option. Here is an example:

_______________________________________________
Bootlin quarterly newsletter
Unsubscribe: %(user_optionsurl)s?password=%(user_password)s
Archives: https://lists.bootlin.com/pipermail/newsletter/

Beware that sending custom e-mails to each recipient will increase the load on your server. You may not want to do this on mailing lists with great numbers of subscribers.

Useful resources

The below ressources were useful to prepare this HOWTO document:

Don’t hesitate to ask questions and give feedback by leaving a reply below.

Bootlin at the Android Builders Summit and the Embedded Linux Conference: one talk and video recording

A good part of the Bootlin team will be in San Francisco (actually, not in San Francisco, but in the Bay Area) from February, 13th to 17th for the Android Builders Summit and the Embedded Linux Conference.

Android Builders Summit 2012
Android Builders Summit 2012

The Android Builders Summit is the second edition of this conference dedicated to Android system development (and not application development). Compared to last year, the conference has been extended to three parallel tracks during two days. There are many talks about Android customization, Android internals, Android porting, usage of Android in specific markets (medical devices, vehicle infotainment), etc. A lot of useful talks for developers working at the Android system level.

Embedded Linux Conference 2012
Embedded Linux Conference 2012

The Embedded Linux Conference is now a well-established conference. Again for this 2012 edition, there will be three parallel tracks during three days. There will be talks about many, many topics: performance and optimization, power management, build systems, drivers for various types of devices, multimedia, ARM kernel support and much more.

I will be giving a talk about Buildroot: A Nice, Simple and Efficient Embedded Linux Build System on the second day of the conference. The aim of the talk is to give a status on where Buildroot is, three years after a maintainer was chosen and a big clean up work was started. The project has changed a lot compared to its state three years ago, so I thought it would be nice to make a status on where Buildroot and where it is going.

With my colleagues Grégory Clément and Maxime Ripard, we will also record all the talks from the Embedded Linux Conference in order to put the videos online, freely available, after the conference, as we have done for many past conferences.

We hope to meet you in San Francisco for the Android Builders Summit and the Embedded Linux Conference!

Announcing our “Android System Development” training

Android RobotFor multiple years, Bootlin has provided two typical training courses for embedded Linux developers: an Embedded Linux system development course that focuses on the basics for embedded Linux development (bootloader and kernel configuration, compiling and usage, system integration and build systems, cross-compiling, filesystems, application development and debugging, etc.) and an embedded Linux kernel and driver development course that focuses on kernel and driver development (kernel APIs for drivers, character drivers, device model, power management, kernel porting, etc.). In total, we have given dozens of editions of these sessions in multiple locations all around the world. We have kept our commitment to release all the training materials under a free license (the Creative Commons CC-BY-SA license), and they are therefore freely accessible for anyone at /docs/.

We are now announcing a new course, called Android System Development. It is a four day training session that targets engineers who need to develop embedded systems with Google Android.

Through theory and practical labs, the course makes you familiar with compiling and booting Android, with adapting Android to support a new embedded board (assuming that it is already supported by the Linux kernel), and with building a real system through accessing specific hardware, customizing the filesystem and using debugging techniques. See the complete agenda. The training materials for this session will also be made available under the same Creative Commons CC-BY-SA license.

If you are interested in this training session, you can:

  • Join the public session organized in Toulouse, France, on June 11-14, 2012.
  • Order an on-site training session to be held at your location. See registration details.

This training course will be given by our engineer Maxime Ripard who has gained Android experience by working at Archos on Android tablets, by making Android work on multiple TI OMAP3 based platforms and also by participating to the Android Builders Summit conference.

Do not hesitate to contact us for further details about this new training course.

Bootlin at FOSDEM: two talks and video recording

I'm going to FOSDEMAs usual, Bootlin will again be present at the FOSDEM conference in Brussels, on February, 4th and February 5th. We will of course mostly be attending the Embedded DevRoom, with multiple talks around development in the embedded space.

We will also be giving two talks this year:

  • My colleague Maxime Ripard will be giving a talk about IIO, a new subsystem for I/O devices. In short, IIO is a new subsystem in the kernel to write drivers for devices like Analog-to-Digital converters. Maxime has worked on a driver inside the IIO subsystem for the internal ADCs of the AT91 processors from Atmel, and will base his talk mostly on the experience developing this driver. This talk will take place on Saturday, 12:00 AM to 1:00 PM in the Lameere room.
  • I will be giving a talk on Using Qt for non-graphical applications. It is a talk that has already been given at the Embedded Linux Conference Europe, but the audience of FOSDEM and ELCE being quite different, we have chosen to propose it for FOSDEM as well, and it got accepted. This talk will take place on Sunday, 1:00 PM to 2:00 PM in the Lameere room.

There are also other talks that are worth noting: a SoC power management talk from Jean Pihet who works on OMAP power management support in the Linux kernel, a talk about OpenCores and OpenRISC, a talk about Safe Upgrade of Embedded Systems by Arnout Vandecappelle, who contributes a lot to Buildroot, and also other talks about OpenWRT, Yocto, licensing issues in Android, the EFL libraries, and more.

We will also be carrying our camcorder to video record those talks. We are trying to see with the FOSDEM organization team if it possible to record the audio directly from the room sound systems in order to provide better audio quality in our videos.

If you happen to be at FOSDEM, we’d be very happy to meet you!

Buildroot Developer Day, Brussels edition

BuildrootAround each FOSDEM conference and Embedded Linux Conference Europe event, we have been organizing a Buildroot Developer Day for a few years, in order to gather some developers and users of the Buildroot build system, in order to discuss the development of Buildroot, its features, development process, design, and more.

In Prague at the last Embedded Linux Conference Europe in October 2011, we had a very interesting meeting that gathered developers from other build systems (OE-lite, OpenBricks and PXTdist), and we published a report of this meeting.

The next Buildroot Developer Day will take place on Friday, 3rd February, just before the FOSDEM conference, in Brussels. This is the first meeting that will gather such a number of Buildroot developers: Peter Korsgaard (Buildroot maintainer), Arnout Vandecapelle (developer from Essensium/Mind, who has been contributing a lot to Buildroot lately), Thomas De Schampheleire (also a big contributor in the last year or so), Luca Ceresoli, Yann E. Morin (developer of Crosstool-NG), my colleague Maxime Ripard (who contributed package enhancements and improvements of the package infrastructure) and myself.

This meeting is open to all Buildroot developers and users, and will take place in a location easily accessible in the center of Brussels. Do not hesitate to contact me at thomas.petazzoni@bootlin.com if you want to take part to this meeting.

New quarterly newsletter: 2011 report, best wishes and 2012 plans

The Bootlin team wishes you a Happy New Year 2012 and all the best for your professional and personal projects. We are taking this opportunity to give some news about Bootlin.

In 2011, Bootlin has:

Worked on multiple development projects for various customers. Amongst the most important ones:

  • development of an embedded Linux system and Qt-based application for a RFID/GSM device based on the AT91 ARM processor
  • boot time reduction on a MIPS-based point-of-sale system, by improving the embedded Linux system integration
  • development of an embedded Linux system for an AT91-based device for the medical field (kernel and bootloader adaptation, system integration, application porting)
  • porting of the PREEMPT_RT patch set to the 2.6.32 kernel delivered by Texas Instruments
  • developed the driver for the Analog to Digital converters built-in the AT91 processors
  • conducted a real-time performance analysis of the PREEMPT_RT and Xenomai solutions on AT91 based processors
  • developed an Ubuntu-based embedded system on a BeagleBoard, for image acquisition and analysis with OpenCV
  • boot time reduction on an i.MX-based device, with major bootloader modifications
  • developed a demonstration system for a racing car control panel on a AT91-based device, with a Qt graphical application

Helped customers solve various embedded Linux related problems, through the support provided by Bootlin engineers

Contributed to various open-source projects:

  • 167 patches to the Buildroot build system
  • 6 patches to the Linux kernel, and more are coming with the mainlining of our AT91 ADC driver
  • 6 patches to the Barebox bootloader
  • 4 patches to the U-Boot bootloader
  • 3 patches to the LTT-ng project

Given multiple sessions of our Embedded Linux system development and Linux kernel and driver development courses. The materials of these courses are being constantly updated and are still freely available under a Creative Commons license.

Prepared materials for a new Android system development course. A four days training session to understand the Android system architecture, how to build and customize an Android system for a given hardware platform, how to extend the Android platform to take new hardware devices into account. A first public session will be organized in June in Toulouse.

Switched the hardware platform used in our Embedded Linux system development course from the aging Calao USB-A9263 platform (AT91-based) to the much more powerful IGEPv2 platform from ISEE (OMAP3-based), offering more possibilities to improve our course.

Hired a new engineer, Maxime Ripard, with Android and embedded Linux experience, and created a new office in Toulouse, France.

Moved its headquarters to Orange, France. While we remain reasonably close the Nice area, where we started, we get closer to other parts of France.

Given two presentations at the Embedded Linux Conference Europe in Prague (Using Buildroot for real projects and Qt for non-graphical applications), gave one presentation on boot time reduction at the GENIVI meeting in Dublin, and gave five editions of an embedded Linux introduction seminar in France.

Attended multiple conferences, for which the Bootlin team also recorded and published videos of the talks:

Participated to the development of the community of Linaro, an engineering organization working on improving Linux on the ARM platform. In addition to making sure that Linaro has all the infrastructure required to nurture a community of developers and users, we also supported Linaro release users on AskLinaro.

In 2012, we expect to:

Work on more development projects in the field of kernel porting, boot time reduction, power management and embedded Linux system integration.

Announce several new training sessions:

  • Git training. A two days training session to clearly understand how to use the Git distributed version control system, both for internal projects and for contribution to open-source projects.
  • Advanced Buildroot training. A three days training session to get a clear and detailed understanding of the Buildroot embedded Linux build system: how to add new packages, how to customize it to generate the embedded Linux system for a given hardware platform.

As we are currently preparing those courses, we are definitely interested in having feedback. Do not hesitate to contact us with your ideas and needs about those topics.

Switch our Linux kernel and driver development course to an OMAP3-based platform, and expand it to the development of a driver for an I2C-attached device.

Convert our training materials to a text source format (LaTeX), and maintain them in a public git tree, making it easier to contribute to them and to follow changes between between versions.

Participate to multiple conferences. Bootlin will be present at the FOSDEM in Brussels in February, at the Android Builders Summit and the Embedded Linux Conference in San Francisco in February, and also at the Embedded Linux Conference Europe in Barcelona in October. This participation to conferences allows Bootlin engineers to remain up-to-date with the latest developments in the embedded Linux area and to create useful contacts in the community.

You can follow Bootlin news by reading our blog (24 articles in 2011) and by following our quick news on Twitter.

Bootlin remains available to help you in your embedded Linux projects, either through its development and support services or through its training sessions. Do not hesitate to contact us!

Best regards, and again, Happy New Year 2012!

Gregory, Maria, Maxime, Michael and Thomas – Bootlin

mkenvimage: a tool to generate a U-Boot environment binary image

Many embedded devices these days use the U-Boot bootloader. This bootloader stores its configuration into an area of the flash called the environment that can be manipulated from within U-Boot using the printenv, setenv and saveenv commands, or from Linux using the fw_printenv and fw_setenv userspace utilities provided with the U-Boot source code.

This environment is typically stored in a specific flash location, defined in the board configuration header in U-Boot. The environment is basically stored as a sequence of null-terminated strings, with a little header containing a checksum at the beginning.

While this environment can easily be manipulated from U-Boot or from Linux using the above mentioned commands, it is sometimes desirable to be able to generate a binary image of an environment that can be directly flashed next to the bootloader, kernel and root filesystem into the device’s flash memory. For example, on AT91 devices, the SAM-BA utility provided by Atmel is capable of completely reflashing an AT91 based system connected through the serial port of the USB device port. Or, in factory, initial flashing of devices typically takes place either through specific CPU monitors, or through a JTAG interface. For all of these cases, having a binary environment image is desirable.

David Wagner, who has been an intern with us at Bootlin from April to September 2011, has written a utility called mkenvimage which just does this: generate a valid binary environment image from a text file describing the key=value pairs of the environment. This utility has been merged into the U-Boot Git repository (see the commit) and will therefore be part of the next U-Boot release.

With mkenvimage you can write a text file uboot-env.txt describing the environment, like:

bootargs=console=ttyS0,115200
bootcmd=tftp 22000000 uImage; bootm
[...]

Then use mkenvimage as follows:

./tools/mkenvimage -s 0x4200 -o uboot-env.bin uboot-env.txt

The -s option allows to specify the size of the image to create. It must match the size of the flash area reserved for the U-Boot environment. Another option worth having in mind is -r, which must be used when there are two copies of the environment stored in the flash thanks to the CONFIG_ENV_ADDR_REDUND and CONFIG_ENV_SIZE_REDUND. Unfortunately, U-Boot has chosen to have a different environment layout in those two cases, so you must tell mkenvimage whether you’re using a redundant environment or a single environment.

This utility has proven to be really useful, as it allows to automatically reflash a device with an environment know to work. It also allows to very easily generate a different environment image per-device, for example to contain the device MAC address and/or the device serial number.