My way of choosing email programs

This is just another article about how to get your emails managed in a graceful way.

Here is my former way of dealing with them: I choose Gmail as my primary email service provider, and use the online Gmail web page for managing most of the emails – view and move to trash. As for emails need my reply or participate, I use Thunderbird 3 + Enigmail with Gmail’s IMAP support, so I can GPG sign my outbound emails and verify signatures of inbound ones from other people. I keep emails on Gmail server and have a backup copy on my PC from Thunderbird’s IMAP sync.

Yes, everything looked good in the days I have only a not-too-much amount of emails and I was happy with all above worked well for a long period. But a bad thing appears while my mailbox size is growing quite fast in the near half year: Thunderbird’s speed is lower and lower, the local Mbox file is larger day by day. Personally I don’t think Mbox format with big amount of emails stored in a single file is so reliable because when there is something wrong, everything soon follow.

My initial thought of turning to another way is, I need to find a better way for managing large amount of growing emails, and three overall requirements are listed below:

  • 1. A not-so-rare solution. I am not familiar with how email system runs so a popular solution can help me find essential documents when I run into trouble.
  • 2. Good stability. Stability is always a key topic when people are finding a solution for their deployment. Even though I am just a desktop user who are looking for a personal way of dealing things, I’d like to have a stable platform to make my life easier.
  • 3. A flexible (and maybe very customizable) way. Flexibility is an important thing when you use a *NIX platform or even all the time; customizability is another great thing once you are willing to pay your hours on making everything just fits your special taste.
  • Next, search the Internet and we can surely found Mutt should be counted in our list of choices, and there are several different usages. Before comparing among them, I need to say something about the email system first.

    There are three key elements in an email system: MUA (Mail User Agent), MTA (Mail Transfer Agent) and MDA (Mail Delivery Agent):

  • 1. An MUA is always the client software that you will face directly everyday – read, write and manage your messages with it.
  • 2. An MTA is a big concept that includes most part of the sending and receiving emails among servers, including but not limited to the services that we may know: SMTP, IMAP, POP3, etc.
  • 3. An MDA is a program deal with user’s mail delivery to their specific mail spool on a particular system; its work are usually processing emails received by MTA and figuring out whether the email is a suspected spam and drop the email to the user specified place of receiving her/his mails.
  • When we are receiving and sending emails on our local computer (not the web based way since it is actually handled by remote server), we need all of the three parts. We get emails from remote server’s inbox with MTA; filter and deliver the emails to our email spool with MDA; read, write and manage emails with MUA; and send emails with a sending MTA. As for the local storage format, there is Mbox that I don’t like, and Maildir I am interested in. More about storage format will be described in following paragraphs.

    There are many alternative all-in-one solutions like Thunderbird, Evolution, KMail and so on, they combine the functions which are required to manage emails for an end-user and hide them behind. Now I am not finding another all-in-one application so I need to choose every program to take their roles in the route of receiving and sending emails. Since I have decided to use Mutt as my MUA from beginning, I don’t need to be bothered by this topic, again.

    First I need a receiving MTA for fetching emails from Gmail server. There are two popular choices: Fetchmail and Getmail4.

  • 1. Fetchmail is a full-featured, robust, well-documented remote-mail retrieval and forwarding utility written in the C programing language. It is famous of the large user base and Eric S. Raymond was maintaining it from 1996 to 2003.
  • 2. Getmail4 is the 4th version of Getmail – which is designed to get rid of the shortcomings in fetchmail. The program is written in Python.
  • Both of the two supports both IMAP and POP3 protocol that are my choices of receiving emails. I prefer to use IMAP because the protocol is designed to have the functions POP3 has and many other features including sync between server and clients. I don’t really need to have mails synced because the local copy is mostly a backup, and I can do it through web interface or Thunderbird if I really need them to be synced. Setting up an IMAP synced local mail solution is kind of off topic from this article, and you may want to try imapsync if you are sure you really need. Another reason of my choosing IMAP over POP3 is, Gmail’s POP3 implementation has violated the commonly expected behavior and there are much more limitations than IMAP (you can fetch around 200-500 mails per session in POP3, and probably be locked 24 hours if you access it several times in a short period, but when you are a newbie and trying to test your configuration you may exceed the limits).

    Now try them out to find which one is better for me. The configuration files always have your email account and password, so make it only readable by yourself using:
    chmod 0600 /path/to/file
    Here are the configurations for both the programs, username and password are changed to “user” and “passwd”:

  • 1. $HOME/.fetchmailrc:
    #set daemon 600
    #set syslog
    poll "" proto IMAP
    user 'user' password 'passwd'
    mda "/usr/bin/procmail"
    keep ssl
    sslcertpath /etc/ssl/certs
    folder "[Gmail]/All Mail"

    The commented lines are telling fetchmail to work as a daemon and try to fetch emails from server every 600 seconds. Option “defaults” tells fetchmail load its default settings. The following two lines describes the server, protocol, username, password; port and local mapped user can be set as well, but I don’t need it. Next line started by “mda” tells fetchmail it should relay received emails to the specific MDA, now we set it to “/usr/bin/procmail” and will be documented in following paragraphs. “keep” means fetchmail should keep the copy on server after getting the mails, this option is not very useful for Gmail because there are preferences on the web interface that you can choose how the server deal with the delete request from clients. “ssl” stands for use ssl, “sslcertck” for checking whether the cert of server has a valid signature by a CA, and “sslcertpath” sets the path of CA cert’s and we use “/etc/ssl/certs”. “fetchall” tells that fetchmail should fetch all emails on the server rather than only new ones. “folder” tells the exact folder fetchmail should go and check, we set it to “[Gmail]/All Mail”.
  • 2. $HOME/.getmail/getmailrc:
    type = SimpleIMAPSSLRetriever
    server =
    username = user
    password = passwd
    mailboxes = ("[Gmail]/All Mail",)

    type = MDA_external
    path = /usr/bin/procmail

    delete = false
    message_log = ~/.mail/getmail.log
    message_log_syslog = false
    read_all = false
    verbose = 2
    delivered_to = false
    received = false

    Briefly, there are three sections in Getmail4’s configuration file: [retriever] (defines which kind of protocol to use and options related), [destination] (defines where to deliver or pass the emails) and [options] (other options for Getmail4).
    I use “SimpleIMAPSSLRetriever” type of retriever and its name suggests that it is used to common SSL enabled IMAP protocol, which meets our needs. Other fields in this section are easy to understand, the last one defines which folder the program should check, and don’t forget the last comma (,) if you only have one mailbox to be listed there. For destination, I was planning (finally I changed my mind, read on) to use procmail so I have a “MDA_external” type destination, and point the path to “/usr/bin/procmail”. In the last section, “delete” defines whether mails should be deleted from remote server after being received, as said in Fetchmail configuration’s explanation, it is not really necessary for Gmail users, “message_log” defines where the log of retrieving every message should go and “message_log_syslog” is the switch whether message retrieve info should be logged to syslog. “read_all” should not be set to true because it is saying that ignoring what have been fetched and get everything every time Getmail4 runs. “verbose” controls log verbosity: if set to 2, print messages about each of its actions; if set to 1, print messages about retrieving and deleting messages (only); if set to 0, ll only print warnings and errors. “delivered_to” and “received” tells Getmail4 whether it should add those two fields to retrieved emails header, personally like them to be set to “no”.

  • Before we start our test, we need to setup our MDA – Procmail. It is the time for me to tell you why I prefer Maildir local storage format to Mbox used by Thunderbird. In Maildir, every email is stored in a single file, and a real folder contains all the files in a “folder” on remote server. The email files are in plain text. There are three most important benefits from Maildir:

  • 1. No need to lock the folder as in Mbox, Maildir stores emails in single plain text files that can accessed by multiple programs at the same time and can be easily maintained by scripts.
  • 2. The maintenance of storage now depends on your file system rather than an email client. Nowadays file systems on *NIX systems like Ext3/4, Reiserfs, Btrfs are really stronger than ever before and is more reliable than a client program, because a file system is designed to maintain files but maintenance of email storage is only one of the functions of a rather smaller project comparing with file system designing. For the worst situation our data get damaged while the mail program is processing things, we get failed for some of the messages in Maildir, but the whole folder would be corrupted in Mbox.
  • 3. Mailboxes in Maildir format can be used through a network file system (like NFS), but Mbox cannot.
  • And here are disadvantages:

  • 1. Maildir is not supported by many client software while Mbox is universally supported.
  • 2. Some filesystems may not efficiently handle a large number of small files (like XFS).
  • 3. Searching text is not as fast as Mbox. If we want to speed up the search process, a helper program with cache is needed.
  • As I can search the text of email within web interface of Gmail, I don’t need to care too much about the searching disadvantages. I am using Ext4 and is strong enough to handle thousands of small files in one directory. At last I am choosing Mutt which supports Maildir very well. I’d like to take the advantages of it now.

    Here is my configuration of Procmail, $HOME/.procmailrc:

    Don’t forget the slash after “$HOME/.mail/inbox/”, if you lose it Procmail will use Mbox, and you if add it, Maildir instead.

    So we can kick off our test now, keep in mind that neither Fethmail nor Getmail4 nor Procmail need root privilege, just run them in your account:
    $ fetchmail -v
    $ getmail
    I have to admit my prediction of result is totally wrong. I thought Fetchmail is used by many many people and is written in the efficient C, Getmail4 only has a smaller user base and is written in Python which may take more resource on many cases. But the result tells me, under the current configuration, neither of them work for me: Fetchmail fails to fetch around 1/10 of my attachments and only get 0.x KB for a 5MB+ email; Getmail4 stuck when fetching mails larger than 5MB.

    What a hell! But I am not stopping because there is another way – Getmail4 is designed to have some MDA functions built-in, so it can deliver messages directly to Maildir or Mbox format for user. It is time to say I like Gmail’s excellent spam filtering feature so that I don’t need to pay so much time on setting up a spam filter with Procmail or Maildrop, and a simple delivery is okay. Now I change the [destination] section of $HOME/.getmail/getmailrc to:
    type = Maildir
    path = ~/.mail/inbox/

    and run again:
    $ getmail
    Great this time and all my emails are retrieved successfully after a long time’s wait (just leave it here and move on other stuff).

    After choosing a suitable receiving MTA (my choice is with MDA built-in), I still need a sending MTA. There are several popular choices, for light-weight ones: msmtp, esmtp; for powerful ones: exim4, postfix and qmail. The last three ones are run as root daemon, designed to be full replacement of the traditional sendmail. Usually we don’t need such big things for a daily purpose, and they are really worth considering if you would like to run a *real* MTA that can exchange emails with other servers.
    Either msmtp or esmtp is designed to work as an agent to forward local email to a real MTA server supports SMTP protocol. Currently msmtp is more welcomed, but the feature list is shorter than esmtp. After a detailed check, that esmtp is not maintained now, so I choose msmtp. Here is my configuration, $HOME/.msmtprc:
    tls on
    tls_starttls on
    tls_certcheck on
    tls_trust_file /etc/ssl/certs/ca-certificates.crt
    logfile ~/.mail/msmtp.log

    account default
    port 587
    auth on
    user user
    password passwd

    There is your username and password in this file, so follow the instruction before to change this file to 0600 mode. “tls”, “tls_starttls” and “tls_certcheck” tells msmtp to use STARTTLS for encryption, and check for validation of the cert.

    Finally, I go to the key part – Mutt. Here is some essential lines from my $HOME/.muttrc:
    ignore *
    unignore From Subject Lines
    hdr_order From Subject Lines

    set index_format="%[%b-%d] %?X?%X& ? %-2e %-18.18L [%4c] %s"
    set status_on_top=yes

    set editor="vim -c 'norm O'"

    set sendmail="/usr/bin/msmtp"
    set sendmail_wait = 5

    set mbox_type=Maildir
    set folder="~/.mail"
    set mask="!^\\.[^.]"
    set mbox="+inbox"
    set record="+inbox"
    set postponed="+inbox"
    set spoolfile="~/.mail/inbox/"
    set trash="~/.mail/trash/"
    set maildir_trash=no

    set quit=yes
    set move=no
    set beep_new=yes
    set check_new=yes
    set recall=no
    set resolve=yes
    set allow_8bit
    set charset="utf-8"
    set rfc2047_parameters=yes

    set include=yes
    set indent_str="> "
    set mime_forward
    set mime_forward_rest
    set fast_reply
    unset metoo
    unset reply_self
    set reply_regexp="^(re([\[0-9\]+])*|aw|回复)(:[ \t]|:)"
    set quote_regexp="^( {0,4}-?[>|:]| {0,4}[a-z0-9]+[>|]+)+"

    set from='Name Last '
    set use_from
    set envelope_from=yes
    set realname='First Last'

    bind index gg first-entry
    bind index G last-entry
    bind index \cf next-page
    bind index \cb previous-page
    bind index ,g group-reply
    bind pager j next-line
    bind pager k previous-line
    bind pager previous-line
    bind pager next-line
    bind pager gg top
    bind pager G bottom

    color hdrdefault black default
    color quoted red default
    color signature brightblack default
    color indicator brightwhite red
    color attachment black default
    color error red default
    color message blue default
    color search brightwhite magenta
    color status brightyellow blue
    color tree red default
    color normal blue default
    color tilde green default
    color bold brightyellow default
    color markers red default

    Thanks to Roy L Zuo (roylzuo at gmail dot com) for great aid! There are much more lines in my mutt configuration, and the colors are suitable for white background.

    In conclusion, here are my choice:
    Mutt + Getmail4 + Msmtp

    Covert GBK/BIG5 encoded ID3 tag to UTF8

    For Chinese Linux users, we may meet many mp3 files with GBK/BIG5 encoded ID3 tags, which are very time consuming to change them to UTF8 one by one.

    Before using the following solution, please confirm that you won’t use media players on Windows that cannot handle UTF8 encoded ID3 tags correctly, and you won’t use MP3 players that don’t support it.

    Firstly, install python-mutagen package. For Debian/Ubuntu, use:
    $ sudo aptitude install python-mutagen
    For Fedora and others, probably:
    # yum install python-mutagen

    Secondly go to the directory containing files need to be converted.
    For files with GBK tags:
    find . -iname "*.mp3" -execdir mid3iconv -e GBK {} \;
    For files with BIG5 tags:
    find . -iname "*.mp3" -execdir mid3iconv -e BIG5 {} \;
    There is a good thing that the program could check if the encoding we selected is suitable, so when we convert GBK encoded files, the BIG5 ones won’t be changed. But please don’t use GB18030 option because it will cause problem when the file aren’t really GB18030 encoded.

    If you need to edit tag text yourself, try easytag to help you. :P

    Ubuntu China LoCo Team resigning and nomination meeting

    Yesterday, we held a meeting about Ubuntu China LoCo Team resigning and nomination, and talked a lot about our LoCo team’s future activities.

    Meeting minutes:
    1.What is Ubuntu loco contact? [happyaron]
    2.Vote for new loco contact, and the new contact decided is Eleanor Chen. (10 Pros, 0 Cons, 4 not voted)
    3.Discussed about FullCircle China team’s work.
    4.Made decisions about participating the Ubuntu Global Jam, and proposed for opening a classroom for teaching people who are interested in becoming an MOTU.
    5.All participants agree with organizing more community activities, but not get a conclusion on when is the best time for starting a Ubuntu user group at Beijing.

    Here is the meeting log (Chinese):

    Covert flac to mp3

    Some time we have to use mp3 format because our moveable devices commonly don’t support flac/ogg, thus I need to convert a .flac file to mp3. It’s very simple:
    First of all, install flac and lame package from your distribution’s repository.
    Then, using the following command to actually covert the file.
    1.flac -d filename.flac
    This will output an filename.wav in the same directory.
    2.lame filename.wav
    Now a fresh filename.mp3 is lying in the same directory.


    Lucid release party in Beijing!

    Ubuntu China LoCo Community is going to host a party at 2:00 PM, on May 8th in the Traktirr Russian Restaurant, Beijing, in order to celebrate the release of Ubuntu 10.04 LTS. Everybody, from newbie to hacker, is warmly welcomed. We will invite some people to give lectures about Ubuntu, but what is truly important is, we hope that everybody can participate in the party! We sincerely wish that all of you can have fun here. Remember that there are a cake, some free CDs of Ubuntu 10.04 LTS and other souvenirs waiting for you!

    Simplified Chinese version of Ubuntu Desktop Course

    After a period’s work, Ubuntu Desktop Course has been translated to Simplified Chinese, the content has already adapted to 9.10, most of them are ready for 10.04. We are happy to announce this to public and wish this course can help more people who speak Chinese enjoy and share Free Software.

    HTML view:
    PDF generation still have some problems.

    This work is licensed under Creative Commons 3.0 Attribution No-Commercial Share-Alike .

    Getting NetworkManager work with pppoe connection on Ubuntu 9.10

    When Ubuntu 9.10 releases, pppoe connection via NetworkManager is impossible because some bug in it. So I switched to the traditional but workable way – pppoeconf, now the problem seems to be solved when using nm team PPA, so I plan to turn back.
    But during my process, there are some other problems. Firstly nm cannot handle the connections automatically; secondly we cannot edit connections system wide.

    Here are the correct steps:

    First, add “NetworkManager daily trunk builds for ubuntu” PPA:

    deb karmic main
    deb-src karmic main

    Second, comment out line “exec pppd call dsl-provider” in /etc/ppp/pppoe_on_boot, that is to say disable my previous “pppoe on boot” setting which is configured by pppoeconf.

    Third, rename /etc/network/interfaces to backup file. NetworkManager will only handle connections which haven’t declared in interfaces, if you didn’t any tunning on such file, you can delete it, but backup before doing any change is a good habit, :)

    Forth, edit /usr/share/polkit-1/actions/ , find out the line contains “System policy prevents modification of system settings”, and below it there is a “auth_admin_keep“, change it to “yes“. This will enable you to edit a system wide connection. If you consider this will do harm to your security, then revert the change once you have set up your connection correctly.

    Fifth, reboot your system, because these settings won’t take effects even though you have run “sudo services network-manager restart” and “sudo services networking restart”.

    Now it is working on my system, cheers!

    Come with Karmic!

    Hi planet, I am finally part of Ubunu Member at the point we are releasing Karmic, cheers for myself and everybody!

    Boosting network for your PC – DNS settings

    Do you think your web browser is not so fast as you expected? I decide to write a series of articles to support you speeding up your network. Follow me to tune your settings and get joys surfing the net!

    This article is written on Linux, especially I am using Ubuntu 9.04, but I will try to mention notices for other distributions and platforms.

    This is the first peice in ‘Boosting network for your PC’, the topic is DNS settings. One question, does your ISP ship low quality DNS service?

    At first place, I recommend you to use OpenDNS, whose slogan is ‘Safer, Faster and Smarter’. Let me tell the reason why to turn form your ISP’s service to this one. OpenDNS is a free service, and there is many features other providers cannot give you, such as a smarter site guide, spell checking, and smart cache to keep your query while the requested site’s name server is not working properly, phishing and malware site bloking … Somebody might say we can use client-based tools to do most of them, but do you think it’s a better way to keep the annoyance away from the source? There is more features provided and I haven’t listed one by one, but it worth a try.

    It is not difficult to turn to OpenDNS, set main DNS server to and backup server to on your PC’s connection setting or on your router. Generally we can change it in resolv.conf on a Linux machine, but GNOME users should do it on network-manager if you use it because it will rewrite resolv.conf automatically. For those who use xDSL, edit your DSL setting, you can find IPv4 option tab, change the method to PPPoE Address Only, and input the two DNS server addresses in order “main, backup”.

    If you have a static IP address, then you should register on, add your network’s IP address to your dashboard, then set your own settings for content filtering, enable the SmartCache, block internal IP addresses to avoid DNS Rebinding Attacks, thus these features are not enabled or can be customized. If you use a dynamic IP address, for example, you are a xDSL user, don’t worry about you cannot have your own account and your custom settings. All you need to do more is install a client to update your address to OpenDNS’s server, and enable the dynamic update feature in you dashboard. On Windows and Mac, they provide their client programs, you can just install can configure it easily according to the guide; on Linux/Unix, you can use ‘ddclient’ which is a free software, you can download it from its homepage on, or you may find it just lying in your distribution’s repository. I recommend you to install it as a service, or called daemon. Here is a example for what I have done on my Ubuntu box:
    1. Install ddclient using apt, or synaptic, or whatever package manager you like, here is what I ran in a terminal:

    sudo aptitude install ddclient

    You don’t need to care about what you have chosen on the debconf interface, soon we’ll chage the configuration file.

    2. Configure ddclient via editing its configuration file, use whatever editor you like to edit /etc/ddclient.conf first, for example I run this command:

    sudo vi /etc/ddclient.conf

    Then make your file look like this:


    The first line without # at its start intends to tell ddclient refresh information to the server every 600 seconds if it is running as a daemon. This section is not needed for Ubuntu user in fact, because it is configured in the other file we’ll look at, just a note here for others who many use it. I think 600 is a suitable value for most network, or you can adjust it to meet your actual needs. Update the information too frequently is a waste of resource, having a too long interval time may let your service not working so properly sometimes.

    The second line intends to make every update logged at syslog, you can delete this line if you don’t want to.

    The third line is for turning on ssl support, in other word we can transfer our account information using an encrypted route.

    The forth line is the PID file path, don’t change it unless you know what your are doing exactly.

    The fifth line tells what protocal you use, OpenDNS requires dyndns2 protocal.

    The sixth line is the way you get your current IP address for transferring to the server, the defaut value was “use=if, if=”, than means you can set a net interface such as eth0 here and ddclient will send eth0’s address. When we are using PPPoE or other virtual connection protocal, this configure may be damnned, so we set it using web to check you real IP, is the host provide such service, you can change it to something you like, but I think copy-and-paste this line to your configuration is the best choice.
    The next three lines is the information about the server and your account, you should copy the server address, and fill in your account name and password there. Don’t worry about your account’s security here, the ddclient.conf is only accessable for root user by default, and we have chosen to use ssl at the begging of this configuration.

    The last line is the LABLE you have set in your dashboard, they are something might be Home, Office, etc.

    Next, we should enable daemon mode for ddclient, edit /etc/default/ddclient.conf with root privilege:

    sudo vi /etc/default/ddclient.conf

    Make sure the following lines are just look like what have shown below:


    The first line is whether ddclient does an update while a new ppp conection is establed, this is useful for those who use a xDSL, or other dial-on-demand connections.

    The second line is whether ddclient run as a daemon, we change it to true here. Only run as a daemon, it can perform the set actions automaticlly.
    The last line is how long should it takes from one update to another, I was explained before in the first line of previous configuration file.

    3. Restart ddclient to make everything into effect.

    sudo /etc/init.d/ddclient restart

    We have finished configuring OpenDNS till here, visit to check!

    Second, use DNS cache to speed up the look up action. This is for Linux/Unix users, I don’t know if there is any good replacement for other platforms. I introduce dnsmasq here to do this task.

    1. Install dnsmasq, you can find it in main aspect of distributions‘ repository.

    For Ubuntu, perform the following action:

    sudo aptitude install dnsmasq

    2. Edit /etc/dnsmasq.conf to fit our use:

    sudo vi /etc/dnsmasq.conf

    And make sure the following line is uncommented:


    You can choose to change other settings if you are interested in the file, there is enough comments for you to know what every options sense, but leave it there is a good solution.

    3. Restart dnsmasq

    sudo /etc/init.d/dnsmasq restart

    4. Add to you DNS list. We can edit resolv.conf to make that line at first of ns servers, or change in network-manager.

    5. Restart networking, you don’t need to restart your computer.

    sudo /etc/init.d/networking restart

    So close and re-open your web browser, do you feel web pages load faster?

    Third, tune your firefox(if you use it). Input about:config in your firefox location bar, filter the following items and change their values to ‘true’:


    And change the following item’s value to 8 or 10:


    Add these items to the list via right click and choose ‘Add -> Interger’, note to set the value according to the last number listed below:

    nglayout.initialpaint.delay 0
    content.notify.backoffcount 5
    ui.submenuDelay 0

    If you don’t have IPv6 connection, also change this item’s value to ‘true’:


    Restart your firefox will make it effect.

    In Ubuntu 9.04, the IPv6 support was built in kernel, if you compile kernel by yourself, you can uncheck this feature if you don’t need it.

    This work by Aron Xu is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported.