Tag: howto

HOWTO – Mailing lists with Mailman on Ubuntu 10.04

GNU mailman logoBootlin is not in the system administration business (we offer free and open-source solutions for embedded systems), but we do our best to share whatever experience we acquire, and whatever code we produce.

We configured a KVM virtual machine to run our mailing lists, and we used Ubuntu 10.04, the long term support (LTS) version. Here are instructions based on this experience. You could also use the same instructions to install Ubuntu 10.04 on a tiny, low power ARM board. And if you wish to use a more recent version of Ubuntu, I expect the steps to be very similar.

Here we assume that your domain is example.com and that you install your mailing list software on lists.example.com. It is indeed a good idea to install your mailing list software on a different server. This way, you won’t mess up with your main web and mail servers. Not having CGI scripts running on it will also keep your main mail server more secure.

Install packages

Install the Apache web server. It will allow administrators to configure and manage the lists, and users to subscribe, unsubscribe and tune their subscription settings.

sudo apt-get install apache2

For e-mail delivery, I chose the Postfix MTA (Mail Transport Agent). Exim would have been a good solution too, but I am more familiar with Postfix, which is already in use on our main mail server. 

sudo apt-get install postfix

Choose the Internet site option.

Now install the mailman package:

sudo apt-get install mailman

Select the languages that you want your mailing list interfaces to support. I chose English and French for the moment. To add more languages later, run:

sudo dpkg-reconfigure mailman

Configure the Apache web server

First copy the sample Apache configuration file provided by the mailman package:

cp /etc/mailman/apache.conf /etc/apache2/sites-available/mailman

Now, in /etc/apache2/sites-available/mailman, enable short URLs by enabling:

ScriptAlias /mailman/ /usr/lib/cgi-bin/mailman/

Also modify /etc/mailman/mm_cfg.py:

DEFAULT_URL_PATTERN = 'http://%s/mailman/'

The last step is to enable your mailman site in Apache:

sudo a2ensite mailman
sudo /etc/init.d/apache2 restart

You should now have a new symbolic link in /etc/apache2/sites-enabled/.

Check that the Mailman website works by opening your mailing lists home page: http://lists.example.com/mailman/listinfo. It should look like https://lists.bootlin.com/mailman/listinfo.

Configure postfix

Enable the following line in /etc/mailman/mm_cfg.py:

MTA='Postfix'

Once the MTA is configured, generate Mailman specific aliases for Postfix:

sudo /usr/lib/mailman/bin/genaliases

Now, you need to configure Postfix through its main.cf file. A convenient way to do this is to run the below commands:

sudo postconf -e 'relay_domains = lists.example.com'
sudo postconf -e 'transport_maps = hash:/etc/postfix/transport'
sudo postconf -e 'mailman_destination_recipient_limit = 1'
sudo postconf -e 'alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases'

Also add the following line to /etc/postfix/transport:

lists.bootlin.com      mailman:

and run:

sudo postmap -v /etc/postfix/transport

You won’t have any mail delivery if you forget. I struggled for a few hours before I realized I forgot this setting.

Now, set correct file ownership:

sudo chown root:list /var/lib/mailman/data/aliases
sudo chown root:list /etc/aliases

For logging and debugging e-mail delivery, I recommend to install the sysklogd package. Without it, you won’t have any mail.info, mail.warn and mail.err files in /var/mail/.

To install this package, enable the universe repository if needed (uncomment the lines with universe in the /etc/apt/sources.list file), and run:

sudo apt-get update
sudo apt-get install sysklogd

To avoid having two mail.info and mail.log files with identical contents, edit /etc/syslog.conf and remove the below line:

mail.*                          -/var/log/mail.log

Also create a /etc/logrotate.d/mail file to rotate logs, as in the below example:

/var/log/mail.* {
        daily
        size 10M
        rotate 4
        compress
        missingok
        notifempty
        create 640 root adm
}

Last but not least, restart Postfix:

/etc/init.d/postfix restart

At this point, a good idea is to check that mail delivery works:

sudo apt-get install bsd-mailx
mailx alice@example.com
Subject: test
test
.
Cc:

Note: that’s the line containing only a dot character that allows to terminate the message.

If the receipient doesn’t receive this message, there is an issue in the way your mail server is configured. This could be because the firewall doesn’t allow connections to outside machines through tcp port 25. Anyway, look at the logs in /var/log/mail.* to get a clue. There is no point going on in this howto until you get this fixed.

Creating the mailman site list

Mailman needs a so-called “site list”, which is the list from which password reminders and such are sent out from. The default name for this list list mailman, though you can change this through the MAILMAN_SITE_LIST setting in /etc/mailman/mm_cfg.py.

To create this list, run:

sudo newlist mailman

You will have to answer a few questions like:

Enter the email of the person running the list: postmaster@example.com
Initial newsletter password: xxx
Hit enter to notify newsletter owner...

Choose the password carefully, as crackers will be able to highjack your mailing list if it is too easy to guess.

The next required step is to add the list aliases to /etc/aliases:

# mailman mailing list
mailman:              "|/var/lib/mailman/mail/mailman post mailman"
mailman-admin:        "|/var/lib/mailman/mail/mailman admin mailman"
mailman-bounces:      "|/var/lib/mailman/mail/mailman bounces mailman"
mailman-confirm:      "|/var/lib/mailman/mail/mailman confirm mailman"
mailman-join:         "|/var/lib/mailman/mail/mailman join mailman"
mailman-leave:        "|/var/lib/mailman/mail/mailman leave mailman"
mailman-owner:        "|/var/lib/mailman/mail/mailman owner mailman"
mailman-request:      "|/var/lib/mailman/mail/mailman request mailman"
mailman-subscribe:    "|/var/lib/mailman/mail/mailman subscribe mailman"
mailman-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe mailman"

In addition to delivering e-mail to the mailing lists, these aliases also allow to subscribe and unsubscribe by writing to special e-mail addresses.

You also need to run the newaliases command, without which there is no e-mail delivery:

sudo newaliases
sudo /etc/init.d/postfix restart
sudo /etc/init.d/mailman restart

Look at the /var/log/mailman/error file for potential issues.

Create regular mailing lists

Regular mailing lists are created in the same way as above: Assuming you want to create a newsletter mailing list. You will need run:

sudo newlist newsletter

Add your new mailing list to /etc/aliases:

# newletter mailing list
newsletter:              "|/var/lib/mailman/mail/mailman post newsletter"
newsletter-admin:        "|/var/lib/mailman/mail/mailman admin newsletter"
newsletter-bounces:      "|/var/lib/mailman/mail/mailman bounces newsletter"
newsletter-confirm:      "|/var/lib/mailman/mail/mailman confirm newsletter"
newsletter-join:         "|/var/lib/mailman/mail/mailman join newsletter"
newsletter-leave:        "|/var/lib/mailman/mail/mailman leave newsletter"
newsletter-owner:        "|/var/lib/mailman/mail/mailman owner newsletter"
newsletter-request:      "|/var/lib/mailman/mail/mailman request newsletter"
newsletter-subscribe:    "|/var/lib/mailman/mail/mailman subscribe newsletter"
newsletter-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe newsletter"

Then, run the usual commands:

sudo newaliases
sudo /etc/init.d/postfix restart
sudo /etc/init.d/mailman restart

Configuring your lists

The easiest way to configure your lists and add members is to open the http://lists.example.com.com/mailman/listinfo URL with a browser.

A few things are also possible from the command line. For example, you can add a member as follows:

echo "alice@example.com" > /tmp/foo
sudo add_members -r /tmp/foo newsletter

Enabling archives

Default file permissions are not completely ready to support mailing list archives:

sudo chown -R root:list /var/lib/mailman/archives
sudo chmod o+rX /var/lib/mailman/archives/private

The second line allows the webserver to access the archives. Note that these settings were forgotten in the official Ubuntu documentation.

Settings for newsletter mailing lists

At Bootlin, we also use Mailman to deliver our newsletters. Mailman provides recipients with an easy mechanism to subscribe by themselves and unsubscribe whenever they want.

To make it even easier to unsubscribe from a newsletter, Mailman can add a special footer to each recipient, with a custom URL that allows to unsubscribe without having to remember one’s password.

To enable this feature, you have to enable the following line in /etc/mailman/mm_cfg.py:

# Extra options
# Allow to personalize each message
# (useful to provide a password-less unsubscribe link)
OWNERS_CAN_ENABLE_PERSONALIZATION = 1

Restart Mailman (/etc/init.d/mailman restart) and go to the administrative interface for your list. Under Non digest options, you will then be able to set the Should Mailman personalize each non-digest delivery? option to Full Personalization.

Then, you can set a custom footer for each recipient in the Footer added to mail sent to regular list members option. Here is an example:

_______________________________________________
Bootlin quarterly newsletter
Unsubscribe: %(user_optionsurl)s?password=%(user_password)s
Archives: https://lists.bootlin.com/pipermail/newsletter/

Beware that sending custom e-mails to each recipient will increase the load on your server. You may not want to do this on mailing lists with great numbers of subscribers.

Useful resources

The below ressources were useful to prepare this HOWTO document:

Don’t hesitate to ask questions and give feedback by leaving a reply below.

Ogg/Theora video mini howto

How to make your own Ogg/Theora videos

Here is how we created the free conference videos we are sharing with you.

Our goal is to show you that it is very easy and pretty cheap to create Ogg/Theora videos using only Free Software tools. It would be great if more people shared what they experience, in particular when they attend interesting presentations!

License

Creative commons

Copyright 2006-2008, Bootlin.
This mini-howto is released under the terms of the Creative Commons Attribution-ShareAlike 2.5 license.

Requirements

A mini-DV camcorder.

Such a device costs approximately 500 US dollars / euros. Mini-dv tapes cost about 5 US dollars / euros.

Note that other devices may be used, such as DVD or harddisk camcorders.

Harddisk camcorders are not a very good solution, because video is stored with a high compression rate (MPEG-4 format). You will not get the best results if you encode from MPEG-4 to Theora, because you will be using low bitrate input compressed with another codec.

A DVD camcorder is fine (MPEG-2 compression), because the input quality would be much better. The best is still DV input, which has very little compression, and allows to get the best of the Theora codec.

Camcorder accessories

A tripod is a must-have. Without one, your image will not be very stable (even with image stabilization), and above all, you will be exhausted after one hour.

An external microphone is nice to have, but not mandatory at all. You still
get pretty good quality audio with the built-in one. So, if you are satisfied
by the audio that you get, you do not have to buy such a microphone. However,
the best solution for top quality audio is to connect you audio input to the
room audio system (if any, and if the speaker is using a microphone).

Computer connectivity

You need a GNU/Linux computer with FireWire input (aka IEEE 1394 or iLink).
If you have a notebook with a PCMCIA adaptor, you best option is to get a
FireWire PCMCIA card which doesn’t need any special driver. This should mean
that it is compliant with the 1394 OHCI standard, which is fully supported by
Linux. Note that recent distributions (at least Fedora Core) automatically
load the right drivers when such a card is plugged in.

It may be possible to use USB connectivity too to get the video from the
camcorder. We just do not know how yet. Any resources are welcome!

Storage

The DV files are huge (roughly 15 GB per hour). As intermediate processing
steps are used in our flow, intermediate files of similar size will be
created. Hence, you will need at least 30 GB of free space to process 1 video.
Anyway, it’s much better to have 100 GB or more to store and process several
videos in a row. For notebook owners, external hard drives (typically
high-speed USB 2.0) are your friends.

An external microphone is nice to have, but not mandatory at all. You still get pretty good quality audio with the built-in one. So, if you are satisfied by the audio that you get, you do not have to buy such a microphone. However, the best solution for top quality audio is to connect you audio input to the room audio system (if any, and if the speaker is using a microphone).

Shooting the video

Before or right after filming, make sure that you ask the speaker(s) for permission to publish the video! Make sure you mention the license that you are going to use.

Video capture

Connect the camera to the computer with the FireWire cable.

If you are using a PCMCIA FireWire adaptor, all the modules should have been loaded automatically at module load time.

If you have a legacy FireWire input, you may have to do a few things by hand (logged as root):

modprobe dv1394
chmod a+rw /dev/dv1394/0

You will now use dvgrab
to get the video through the FireWire link and save it to a file. This tool is shipped by most distributions.

dvgrab --size 0 --format raw <output-file-prefix>

Note: --size 0 means that the output file is not split into many smaller ones, when they exceed a given size.

Now that you’re done, let’s assume that you created a video.dv file.

Video trimming

When you read reused tapes, it’s hard to avoid video frames from the previous recordings at the begining or at the end. Before compressing, you first have to trim out the unwanted frames.

This is pretty easy to do with the kino tool, available in all recent distros.

Make sure you export the trimmed video in DV format, to avoid losing quality.

We will soon post kino usage screenshots on this page, to get you started faster with kino.

Quick Ogg/Theora generation

That’s very easy to do. Get the latest version of the ffmpeg2theora package.

ffmpeg2theora -o video.ogv video.dv

You’re done!

You can use the -v and -a parameters to control video and audio quality. The defaults (5 and 2) should be fine for average quality requirements. With -v 7, we already get very good video quality, but the output file size is roughly double. As far as audio quality is concerned, keep in mind the source quality. Unless your audio input is high quality (audio in directly connected to the conference room sound system), there is no need for high bitrate audio compression (-a setting greater than 4).

Deinterlacing?

If the output video quality is poor, it could be because your video needs deinterlacing. In particular, this happens when you record your video in long play mode. Interlaced video is very easy to identify: you just need to find a sequence with motion (camcorder or character motion). Pause the video and interlaced lines will show up.

So, if you source video is interlaced, use the --deinterlace parameter of ffmpeg2theora:

ffmpeg2theora --deinterlace -o video.ogv video.dv

Denoising the video

Look carefully at the generated Ogg/Theora video. Do you see MPEG-like squares moving on surfaces which shouldn’t change at all (walls, sky, board, etc.)?

If this happens, this means that your original video contained noise. This is very frequent with digital camcorders, in particular in low light conditions (when you amplify a weak signal, noise gets more significant). Such noise, though it is not obvious on the source video, can get amplified in the compression process.

Hence, it’s best to remove noise before compressing, so that pixes in still surfaces do not change at all in the source video. Follow the below instructions and compare the output Ogg/Theora video size. You will find that the output file is smaller that what you got by just running ffmpeg2theora on the raw DV video.

Fortunately ffmpeg2theora now supports denoising filters: we contracted Jan Gerber, its developer, to add such filters to his tool. First make sure you have at least version 0.20 (otherwise, download the latest version).

The implementation is based on ffmpeg / mplayer‘s postproc library. Available filter settings are detailed by ffmpeg2theora --pp help, or can be found by looking for tmpnoise in mplayer’s manual page. Filter settings are not easy to choose, however. For your convenience, here are the settings we chose after multiple experiments: --pp de,tn:256:512:1024. At least with our videos, they produce good quality output without significant side effects.

Ogg/Theora video with metatags

It’s possible and useful to add metainformation (title, author, location, license) to the ogv video files.

This can be done thanks to ffmpeg2theora parameters:

ffmpeg2theora -a 3 -v 7 --pp de,tn:256:512:1024 \
--artist "Michael Opdenacker" --title "Fosdem 2006" \
--date "February 2006" --location "ULB, Brussels, Belgium" \
--organization "Bootlin (https://bootlin.com)" \
--copyright "Copyright 2006, Michael Opdenacker" \
--license "Creative Commons Attribution-ShareAlike 2.5" \
-o video.ogv video.dv

If you need to mass encode several videos in a script, it is now possible to add the metatags by hand after encoding. This can be done with the TagTheora tool.

Going further

Run ffmpeg2theora --help for details about more possibilities like live encoding and streaming.

Thanks

  • To Diego Rondini, for letting us know about TagTheora