Buildroot simplified for users!

Buildroot logoYesterday, a set of patches I’ve authored that aims at simplifying Buildroot for users has been merged into the official version of the project, and will therefore be part of the next stable release (scheduled for November, according to our 3 months release cycle). This work is probably my major contribution to Buildroot, outside of external toolchain support and various fixes here and there. Here are quick details about the improvements brought by these patches :

  • Remove the “project” feature. The project feature removal was the main point of this patch set. This feature, that allows to compile a system for different, but very similar platforms, without recompiling everything from scratch, was rarely used and introduced a lot of complexity in the usage of Buildroot for newcomers. Who hasn’t been confused by this project_build_arm directory? This thing is gone now.
  • Remove the BOARD/LOCAL feature, which duplicates another way of adding support for new targets in Buildroot. This is the kind of feature that has been added at the time Buildroot was basically unmaintained, when nobody was able to say « Hey, but you’re just trying to-reimplement something that already exists »
  • Move all output directories in an output directory. By default, when Buildroot is compiled, it generates several directories in the middle of its source code. Now, with this patch, everything is grouped into an output directory, unless out-of-tree compilation is used, of course (with O=)
  • Remove TOPDIR_PREFIX and TOPDIR_SUFFIX since the same effect could already be done using out-of-tree compilation with O=. Another duplicated feature that should never have reached the tree.
  • Rename the output directories. Now that everything is properly stored in an output directory, it was time to rename the subdirectories to make them more meaningful. So now, we have build where all packages are built, images that contains the final binary images of the root filesystem and the kernel, staging which contains the staging directory (all packages installed with their development headers and libraries), target that contains the root filesystem for the target (without the device files), host that contains the installation of the host tools that Buildroot requires for its execution, stamps that contains the stamp files used by Buildroot to keep track of the compilation progress. Therefore, all directories such as build_ARCH or toolchain_build_ARCH have disappeared.
  • Major documentation update, to of course make sure that our documentation is up-to-date with the latest changes.

Getting all these changes mainlined is really a nice thing. I also have tons of other ideas to improve Buildroot infrastructure, and I’m sure the coming Buildroot Developer Day will be a great opportunity to discuss these.

Faster boot: starting Linux directly from AT91bootstrap

Reducing start-up time looks like one of the most discussed topics nowadays, for both embedded and desktop systems. Typically, the boot process consists of three steps: AT91SAM9263 CPU

  • First-stage bootloader
  • Second-stage bootloader
  • Linux kernel

The first-stage bootloader is often a tiny piece of code whose sole purpose is to bring the hardware in a state where it is able to execute more elaborate programs. On our testing board (CALAO TNY-A9260), it’s a piece of code the CPU stores in internal SRAM and its size is limited to 4Kib, which is a very small amount of space indeed. The second-stage bootloader often provides more advanced features, like downloading the kernel from the network, looking at the contents of the memory, and so on. On our board, this second-stage bootloader is the famous U-Boot.

One way of achieving a faster boot is to simply bypass the second-stage bootloader, and directly boot Linux from the first-stage bootloader. This first-stage bootloader here is AT91bootstrap, which is an open-source bootloader developed by Atmel for their AT91 ARM-based SoCs. While this approach is somewhat static, it’s suitable for production use when the needs are simple (like simply loading a kernel from NAND flash and booting it), and allows to effectively reduce the boot time by not loading U-Boot at all. On our testing board, that saves about 2s.

As we have the source, it’s rather easy to modify AT91bootstrap to suit our needs. To make things easier, we’ll boot using an existing U-Boot uImage. The only requirement is that it should be an uncompressed uImage, like the one automatically generated by make uImage when building the kernel (there’s not much point using such compressed uImage files on ARM anyway, as it is possible to build self-extractible compressed kernels on this platform).

Looking at the (shortened) main.c, the code that actually boots the kernel looks like this:

int main(void)
{
/* ================== 1st step: Hardware Initialization ================= */
/* Performs the hardware initialization */
hw_init();

/* Load from Nandflash in RAM */
load_nandflash(IMG_ADDRESS, IMG_SIZE, JUMP_ADDR);

/* Jump to the Image Address */
return JUMP_ADDR;
}

In the original source code, load_nandflash actually loads the second-stage bootloader, and then jumps directly to JUMP_ADDR (this value can be found in U-Boot as TEXT_BASE, in the board-specific file config.mk. This is the base address from which the program will be executed). Now, if we want to load the kernel directly instead of a second-level bootloader, we need to know a handful of values:

  • the kernel image address (we will reuse IMG_ADDRESS here, but one could
    imagine reading the actual image address from a fixed location in NAND)
  • the kernel size
  • the kernel load address
  • the kernel entry point

The last three values can be extracted from the uImage header. We will not hard-code the kernel size as it was previously the case (using IMG_SIZE), as this would lead to set a maximum size for the image and would force us to copy more data than necessary. All those values are stored as 32 bits bigendian in the header. Looking at the struct image_header declaration from image.h in the uboot-mkimage sources, we can see that the header structure is like this:

typedef struct image_header {
uint32_t    ih_magic;    /* Image Header Magic Number    */
uint32_t    ih_hcrc;    /* Image Header CRC Checksum    */
uint32_t    ih_time;    /* Image Creation Timestamp    */
uint32_t    ih_size;    /* Image Data Size        */
uint32_t    ih_load;    /* Data     Load  Address        */
uint32_t    ih_ep;        /* Entry Point Address        */
uint32_t    ih_dcrc;    /* Image Data CRC Checksum    */
uint8_t        ih_os;        /* Operating System        */
uint8_t        ih_arch;    /* CPU architecture        */
uint8_t        ih_type;    /* Image Type            */
uint8_t        ih_comp;    /* Compression Type        */
uint8_t        ih_name[IH_NMLEN];    /* Image Name        */
} image_header_t;

It’s quite easy to determine where the values we’re looking for actually are in the uImage header.

  • ih_size is the fourth member, hence we can find it at offset 12
  • ih_load and ih_ep are right after ih_size, and therefore can be found at offset 16 and 20.

A first call to load_nandflash is necessary to get those values. As the data we need are contained within the first 32 bytes, that’s all we need to load at first. However, some space is required in memory to actually store the data. The first-stage bootloader is running in internal SRAM, so we can pick any location we want in SDRAM. For the sake of simplicity, we’ll choose PHYS_SDRAM_BASEhere, which we define to the base address of the on-board SDRAM in the CPU address space. Then, a second call will be necessary to load the entire kernel image at the right load address.

Then all we need to do is:

#define be32_to_cpu(a) ((a)[0] << 24 | (a)[1] << 16 | (a)[2] << 8 | (a)[3])
#define PHYS_SDRAM_BASE 0x20000000

int main(void)
{
unsigned char *tmp;
unsigned long jump_addr;
unsigned long load_addr;
unsigned long size;

hw_init();

load_nandflash(IMG_ADDRESS, 0x20, PHYS_SDRAM_BASE);

/* Setup tmp so that we can read the kernel size */
tmp = PHYS_SDRAM_BASE + 12;
size = be32_to_cpu(tmp);

/* Now, load address */
tmp += 4;
load_addr = be32_to_cpu(tmp);

/* And finally, entry point */
tmp += 4;
jump_addr = be32_to_cpu(tmp);

/* Load the actual kernel */
load_nandflash(IMG_ADDRESS, size, load_addr - 0x40);

return jump_addr;
}

Note that the second call to load_nandflash could in theory be replaced by:

load_nandflash(IMG_ADDRESS + 0x40, size + 0x40, load_addr);

However, this will not work. What happens is that load_nandflash starts reading at an address aligned on a page boundary, so even when passing IMG_ADDRESS+0x40 as a first argument, reading will start at IMG_ADDRESS, leading to a failure (writes have to aligned on a page boundary, so it is safe to assume that IMG_ADDRESS is actually correctly aligned).

The above piece of code will silently fail if anything goes wrong, and does no checking at all – indeed, the binary size is very limited and we can’t afford to put more code than what is strictly necessary to boot the kernel.

First Buildroot Developer Day after ELCE, on October 17th in Grenoble, France

Buildroot logoThe first Buildroot Developer Day will take place on Saturday, October 17th in Grenoble, France, just the day after Embedded Linux Conference Europe. This Developer Day aims at allowing Buildroot developers to meet and exchange ideas on the project and its future.

As the number of places is limited, interested candidates are invited to send an e-mail to Peter Korsgaard (jacmet at uclibc dot org) and Thomas Petazzoni (thomas dot petazzoni at free-electrons dot com).

Peter Korsgaard and I are organizing this Developer Day thanks to the sponsorship of Calao Systems (offering the location for the meeting) and Bootlin (offering free lunch to the participants).

See also the announcement of the mailing list and on Buildroot website.

Embedded Linux Conference Europe 2009 program

GrenobleAs we announced a few months ago, Embedded Linux Conference 2009 will take place in Grenoble, France on October 15th and 16th. The list of sessions has just been published online, and as usual, it is a very exciting list of in-depth technical talks.

Because of my involvement in Buildroot, I’m particularly interested in the numerous talks about embedded Linux build systems: Florian Fainelli will talk about OpenWRT (which is not only dedicated to Wifi routers, but is a generic embedded Linux build system, built as a fork of Buildroot), Gordon Hecker on e2factory (a build system I’ve never heard of until now, but conferences are great to discover new tools and projects), Cedric Hombourger on OpenEmbedded (I will be particularly happy to meet Cedric again since I had the chance to work with him six years ago), Marcin Jusziewicz on OpenEmbedded again, Robert Schwebel on PTXdist and finally Alex de Vries on what seems to be a more generic talk about build systems.

Of course, besides build systems, a lot of other topics will be covered. I’ve noted things such as the talk on Canola by Gustavo Barbieri, the boot time presentation by Grégory Clément, the device tree talks by Wolfram Sang and Vitaly Wool, the talk by Alessandro Rubini in order to meet one of the author of Linux Device Drivers, the power management and clock management talks also.

Bootlin will obviously be present during this conference :

  • Michael Opdenacker, my colleague, will give a talk entitled Update on boot time reduction techniques
  • Michael, again, will be the chair of a Small Business BOF, which should allow small companies offering services around embedded Linux to meet and exchange their ideas and experience
  • Finally, I will be the co-chair with Peter Korsgaard of a BOF on Buildroot
  • We will share a booth with our partner Calao Systems during the Minalogic Embedded Systems Exhibition. This will be another opportunity to meet.

I hope to see you in October at ELCE !

Add an introduction sequence to a Theora video

FilmTo produce new HD videos from the Embedded Linux Conference, we needed a new script to add an introduction sequence to the videos we processed.

To make all this automatic, and to suppress the need to go through interactive tools (such as kdenlive) to produce a title with a fade-in / fade-out effect, I created a new theora-intro script. The process got much more complex than I expected, but fortunately, I got a lot of valuable advise and help from the Theora mailing list, and could hide all this complexity inside the script. That’s what scripts are for!

A nice feature. compared to the interactive video editors, is that no extra video processing pass is needed. Thanks to this, even a big video file can be processed within a few minutes. Another useful feature is that the script preserves the Ogg metadata (artist, title, location, copyright…).

So, if you wish to add an introduction sequence to your own videos, whatever their initial format, first convert them to Theora using ffmpeg2theora. Then, create a title image for your video, and then run theora-intro on it.

You can find all useful details on our theora-intro page.

ELC 2009 videos

My Colleague Thomas and I had the privilege to participate to the 2009 edition of the Embedded Linux Conference, which took place in San Francisco, on April 6-8. In spite of the weak economy, this event was once again a success. It attracted major developers from the embedded Linux community, as well as participants from all over the word.

Following the tradition, we are proud to release new videos about this event. These videos were shot by Satoru Ueda and Tim Bird (Sony), and by Thomas Petazzoni and Michael Opdenacker (Bootlin). For the first time, we used an HD camcorder to shoot some of the videos. A higher resolution allows to read the slides projected on the screen. As usual, the videos are released with a Creative Commons Attribution – ShareAlike 3.0 license.

Thomas and I found the following talks particularly interesting:

  • Ubiquitous Linux, by Dirk Hohndel
  • Embedded Linux and Mainline Kernel, by David Woodhouse
  • What are Interrupt Threads and How Do They Work?, by Reece Pollack
  • Visualizing Process Memory, by Matt Mackall
  • KProbes and Systemtap Status, by Tim Bird
  • Deploying LTTng on Exotic Embedded Architectures, by Mathieu Desnoyers
  • Embedded Linux on FPGAs for fun and profit, by Dr John Williams (Petalogix)
  • Linux on Embedded PowerPC porting guide, by Grant Likely
  • Understanding and writing an LLVM Compiler Backend, by Bruno Cardoso Lopes

You may be interested in watching the presentations we made and the BOFs we led:

  • Building Embedded Linux Systems with Buildroot, by Thomas Petazzoni. In these last months, Thomas has made big contributions to this build system.
  • Build tools BOF, by Thomas Petazzoni
  • Update on filesystems for flash storage, by Michael Opdenacker
  • System Size BOF, by Michael Opdenacker

Of course, lots of other talks were very interesting. See the whole list by yourself:

theora-intro

Add an introduction sequence to a Theora video

Introduction

For the needs of producing conference videos, we developed a Python script to add an introduction sequence to a given Ogg/Theora video:

Usage: theora-intro [options] input-video title-image output-video

Adds an introduction to a source Ogg/Theora video, using the given title image

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         Verbose mode
  --artist=ARTIST       Overrides the ARTIST Ogg metadata
  --title=TITLE         Overrides the TITLE Ogg metadata
  --date=DATE           Overrides the DATE Ogg metadata
  --location=LOCATION   Overrides the LOCATION Ogg metadata
  --organization=ORGANIZATION
                        Overrides the ORGANIZATION Ogg metadata
  --copyright=COPYRIGHT
                        Overrides the COPYRIGHT Ogg metadata
  --license=LICENSE     Overrides the LICENSE Ogg metadata
  --contact=CONTACT     Overrides the CONTACT Ogg metadata

Why was this script created?

This script was created especially for people who need to produce multiple videos, typically from the same event. So far, the only way to add an introduction video with fade-in and fade-out was through interactive video editors, such as kdenlive, and this required to encode the video again. Another reason to create a script is that the sequence of commands to run and the amount of data to extract and generate is rather complex. Having a script rather than just a howto thus reduces the risk of errors and mistakes.

Downloads

All the releases of theora-intro can be found here.

How it works

To create an introduction sequence and concatenate it to the input video, theora-intro first needs to collect information from the input video. In particular, it reads the video width, height, the number of frames per seconds, as well as the audio bitrate and sample frequency, which need to be the same in the introduction sequence.

The introduction sequence shows the input title image (scaled), which fades in from a black picture, and eventually fades out to black again. The script actually generates a series of PNG images (using ImageMagick‘s convert utility), and converts this series into a video using ffmpeg2theora. Any input image format should work (JPG, GIF, PNG…), as ImageMagick supports most existing formats.

To be complete, the introduction sequence also needs an audio track. If it didn’t, the output video wouldn’t have any. Therefore, theora-intro generates a silent sample in WAV format, and converts it to Ogg/Vorbis.

The audio and video for the introduction sequence are then merged and concatenated. For some reasons still a bit unclear, the audio tracks of the introduction and input videos need to be resampled, but at last, there is no need to encode the video again. This makes theora-intro much faster than the time it takes to encode the video (just a few minutes even for big video files encoded in several hours).

You can find out more details by reading the code! It shoudn’t be difficult to understand it.

Ogg Metadata

It is useful to produce Ogg/Theora videos with appropriate metadata (artist, title, location, copyright…). If the input video contains such metadata, these metadata are also replicated to the generated video. Note that theora-intro has options to override these metadata when needed, or when there are no such metadata in the input video.

Requirements

This script relies on several software packages and libraries:

In Ubuntu and Debian, you can get the first four packages as follows:

sudo apt-get install vorbis-tools imagemagick ffmpeg2theora oggz-tools

Anyway, if any of the above packages is missing, theora-intro will let you know.

Use in real life

theora-intro was used to produce videos from the 2009 edition of the Embedded Linux Conference.

Known issues

At the moment, videos generated with the latest release still show a few warnings with the ogginfo command:

$ ogginfo elc2009-bird-closing.ogv 
Processing file "elc2009-bird-closing.ogv"...

New logical stream (#1, serial: 376a8b22): type theora
New logical stream (#2, serial: 06fd37a7): type vorbis
Theora headers parsed for stream 1, information follows...
Version: 3.2.1
Vendor: Xiph.Org libTheora I 20081020 3 2 1
Width: 1280
Height: 720
Total image: 1280 by 720, crop offset (0, 0)
Framerate 25/1 (25.00 fps)
Aspect ratio undefined
Colourspace: Rec. ITU-R BT.470-6 Systems B and G (PAL)
Pixel format 4:2:0
Target bitrate: 0 kbps
Nominal quality setting (0-63): 63
User comments section follows...
	TITLE=Tim Bird (Sony)- ELC Closing
	DATE=May 2009
	LOCATION=San Francisco
	ORGANIZATION=Bootlin
	COPYRIGHT=Bootlin
	LICENSE=Creative Commons BY-SA 3.0
	CONTACT=feedback@...
	ENCODER=ffmpeg2theora-0.23
Vorbis headers parsed for stream 2, information follows...
Version: 0
Vendor: Xiph.Org libVorbis I 20070622 (1.2.0)
Channels: 2
Rate: 48000

Nominal bitrate: 48.000000 kb/s
Upper bitrate not set
Lower bitrate not set
User comments section follows...
	ENCODER=oggVideoTools 0.8
Warning: Expected frame 5265, got 5266
Warning: Expected frame 5270, got 5269
Warning: Expected frame 12663, got 12664
Warning: Expected frame 12667, got 12666
Warning: Expected frame 13657, got 13658
Warning: Expected frame 13660, got 13659
Warning: Expected frame 14213, got 14214
Warning: Expected frame 14218, got 14217
Warning: Expected frame 14875, got 14876
Warning: Expected frame 14879, got 14878
Warning: Expected frame 17635, got 17636
Warning: Expected frame 17638, got 17637
Vorbis stream 2:
	Total data length: 4322809 bytes
	Playback length: 11m:57.264s
	Average bitrate: 48.214426 kb/s
Logical stream 2 ended
Theora stream 1:
	Total data length: 49233283 bytes
	Playback length: 11m:57.320s
	Average bitrate: 549.080277 kb/s
Logical stream 1 ended

These warnings don’t seem to create any issue in Ogg/Theora players. They are probably caused by an issue in the Ogg Video Tools, and we have reported this to their maintainer.

On the road to Buildroot 2009.08

Buildroot logoAs I posted back in March, the Buildroot project has begun a new life since January. One of the aspects this new life is the fact that stable releases are delivered every three months. So we had one in February (2009.02), one in May (2009.05) and therefore we’re going to have one release soon in August (2009.08).

Peter Korsgaard, Buildroot’s maintainer, has just released 2009.08-rc1, a good opportunity to look at what’s new in Buildroot since the last release. To highlight Bootlin’sparticipation in the free software community, it is worth noting that in number of patches, I’m the second most important contributor to Buildroot, right after Peter, the maintainer. It’s really nice that I’ve been able to find enough time to contribute at such a level to Buildroot, and hope to be able to dedicate more time in the future to Buildroot, since we have lots of ideas to improve the project.

Back to the 2009.08 release specifically, in the notable features or improvements, we have :

  • Improvement of external toolchain support. I’ve already posted about this specific feature, that I find very important. Several reports from people using this feature have been received since then, which means that it is a useful feature for some of the Buildroot users. So, we have now support for glibc toolchains, the configuration provided to Buildroot is verified against the real toolchain configuration, the copy of the toolchain libraries to the target has been improved (including the C++ standard library) and other minor improvements have been. I’ve been the major contributor of these improvements, together with the reports provided by Buildroot users
  • Integration of the QT-based configurator. Buildroot uses the kconfig system from the kernel to allow its users to configure the different elements of the system to be built. Until now, only the ncurses interface (make menuconfig) was supported. In the 2009.08, the QT interface (make xconfig) will also be supported, adding a little more userfriendliness to Buildroot. This feature was contributed by Alper Yildirim, and I helped in polishing the remaining issues.
  • Support for the Xtensa architecture has been contribued by Maxim Grigoriev, from Tensilica, the company designing the Xtensa architecture.
  • Cleanup of X.org support. the configuration options were clarified, and mandatory dependencies on useless libraries such as libXt or libXaw were removed, some components were upgraded to fix build issues. It’s also a task I contributed to.
  • Of course, toolchain components have been upgraded (GCC 4.4.1, 2.6.30 kernel headers), new packages have been added (bmon, ctorrent, dosfstools, enchant, gst-plugins-bad, iw, libmms, libnl, netstat-nat, ntfsprogs, sdl_gfx, spawn-fcgi), and many more packages have been upgraded (bind, busybox, coreutils, sqlite, directfb, expat, gamin, gnuconfig, haserl, ipsec-tools, classpath, libcurl, libglib2, liblockfile, libpng, libsoup, libxml2, lighttpd, ltp-testsuite, lvm2, matchbox, memstat, gst-plugins-good, gstreamer, libogg, libvorbis, mplayer, neon, openssl, pciutils, php, qt, ruby, sawman, webkit, wpa-supplicant, xdriver_xf86-input-synaptics, xdriver_xf86-video-intel, xlib_libXfont, xlib_libXft, xlib_libXt, xproto_xproto, xserver-xorg, xutil_makedepend, xutil_util-macros)
  • We’re also paying a lot more attention to the bugs reported in our bugtracker than we used to do in the past. Since 2009.05, 43 reported issues have been fixed, and I except that we will fix some more before the 2009.08 release

Shortly after the 2009.08, we will set the goals for the next 2009.11 release, and I’d like to do a lot of cleanup to make Buildroot easier to use and to understand. We’ll see how things go.

In the mean time, if you’re interested in testing and improving Buildroot, don’t hesitate to grab 2009.08-rc1, try it in your conditions with your package set, and report your issues!

Switching between toolchains made easy!

It is quite common to have several toolchains and to switch back and forth between them while doing development. At least, this is something I do a lot when doing Buildroot development and debugging. As I hate typing full paths all the time, I usually put the toolchain bin/ directory into my $PATH variable, so that I can easily access the toolchain binaries. However, it means that everytime you run a new shell or everytime you want to switch from one toolchain to another, you need to modify the PATH variable manually by re-exporting it. Of course, one could easily put all the bin/ directories of all toolchains in the PATH, but that would clutter what is shown when I do arm-TAB-TAB, and that’s not nice.

So, I ended up hacking a few lines of Bash that provide me with two new commands: xtoolsadd, to add a toolchain to my PATH and xtoolsdel, to remove a toolchain from my PATH. These commands work by making the assumption that all toolchains are stored in a common directory. In my case /usr/local/xtools/ contains all the toolchains, one per subdirectory. So I have /usr/local/xtools/arm-unknown-linux-gnu for an ARM glibc-based non-EABI toolchain, or /usr/local/xtools/arm-unknown-linux-uclibcgnueabi for an ARM uClibc-based EABI toolchain).

So, now I can do things such as

xtoolsadd arm-unknown-linux-gnu

or

xtoolsdel arm-unknown-linux-uclibcgnueabi

Because these commands must modify the PATH variable of the current shell, they cannot be implemented as separate shell scripts, so they are in fact implemented as functions in my ~/.bashrc script. And in addition to these functions, I also implemented completion, so when you do xtoolsadd TAB-TAB, it gives you a choice of toolchains, and if you start typing one and press TAB, it will just automatically complete for you. The same thing works with xtoolsdel, of course.

To make this work, here is what you need to put in your ~/.bashrc file:

export XTOOLSDIR=/usr/local/xtools

xtoolsadd() {
    TOOLCHAINDIR=$XTOOLSDIR/$1/bin
    if [ ! -d $TOOLCHAINDIR ] ; then
        echo "Directory $XTOOLSDIR doesn't exist"
    else
        case "$PATH" in
            *"$XTOOLSDIR"*)
                ;;
            *)
                export PATH=$TOOLCHAINDIR:$PATH
        esac
    fi
}

xtoolsdel() {
    TOOLCHAINDIR=$XTOOLSDIR/$1/bin
    NEWPATH=
    found=0
    for i in $(echo $PATH | tr ":" "\n") ; do
        if [ $i == $TOOLCHAINDIR ] ; then
            found=1
        else
            NEWPATH=$NEWPATH:$i
        fi
    done
    if [ $found == 0 ] ; then
        echo "$1 is not in your PATH"
    else
        export PATH=$NEWPATH
    fi
}

_xtoolsadd() {
    cur=${COMP_WORDS[COMP_CWORD]}
    LIST=$(ls -1 $XTOOLSDIR | tr "\n" " ")
    COMPREPLY=( $( compgen -W "$LIST" -- $cur))
}

_xtoolsdel() {
    cur=${COMP_WORDS[COMP_CWORD]}
    LIST=$(echo $PATH | tr ":" "\n" | grep "^$XTOOLSDIR" | sed "s%$XTOOLSDIR/\([^/]*\)/bin%\1%")
    COMPREPLY=( $( compgen -W "$LIST" -- $cur))
}

complete -F _xtoolsadd xtoolsadd
complete -F _xtoolsdel xtoolsdel

The shell code may not be perfect or fully optimized, but it works. Of course, if you have suggestions or questions, don’t hesitate to post comments!