How-to: Alpine and maildir (and offlineimap (and GMail)) — no more maildir patches!

Abstract

If you think non-graphical email clients are efficient or otherwise desirable, and want offline access to your email, this post describes a step forward. In particular, it describes the set-up of offline-accessible mail through the Alpine (formerly Pine, shortly Realpine) email client under (Ubuntu/Debian) GNU/Linux.

Orientation

Finally, I think we can move beyond the dreaded “maildir patch” problems associated with Pine/Alpine/Realpine since it became fully open-source. I would, ultimately, like to see the setup of alpine / mutt / emacs / etc email clients be as easy as GUI clients, at least for those who want a pretty standard setup. For instance, in my case, I want

• to have my email accessible locally, ie “offline”, not just because I am sometimes reading it on a bus but because it’s much faster that way;
• to be able to use fast, keyboard-driven interfaces which nicely integrate my favourite editor into email composition and maybe even navigation;
• to have an interface that parallels what I get on GMail, since I archive all my email accounts there and still sometimes use GMail’s interface for searching email archives. That is, I want my email client to show my GMail labels (as folders) and to add them to messages and to move messages around, just like GMail does.
• to be able to switch between email clients when I like, rather than being tied to one by its own storage format.
• to have the key connections between components be updated at the package level when the underlying daemons etc are updated; for instance, dovecot has changed a couple of times recently, upsetting people’s settings in the offlineimap or alpine configuration. If they are to be seen to work together as a system, the interface between them should ideally be updated automatically.

Overview

The solution is to store mail locally in a Maildir format and to have easy and robust ways for any/each email client to communicate with the local mail repository. That means making public, or offering as pre-fabricated packages, simple setups for the various email clients.

Thanks to Asheesh Laroia, I now have a simple installation that allows alpine to seamlessly use a local IMAP server as a plugin-interface to my email. This makes it easy to swap in or out any other mail client. We aim to bundle a setup of Alpine and Dovecot in order to make this option easy. It is especially useful when combined with Offlineimap, which syncs your local mail repository with whatever is your primary email server. For instance, my setup, used as an example, below, gives me a local, on/off-line Alpine interface to my GMail.

A limitation is that this does not cover the setup for sending messages while offline. Also, the simple setup below includes writing your password as plain text in the offlineimap configuration file.

Setup (start here if you like!):

In the following instructions, replace GMAILUSERNAME with your GMail username or other email service username, and replace NIXUSER with your log-in name on your local machine.

Installing system software (under Ubuntu; requires root access)

Execute the following at a command line:

sudo apt-get install dovecot-common alpine offlineimap dovecot-imapd gnome-schedule

Dovecot (imapd)

Create a new file, for instance, called ~/.auto-dovecot.conf to contain a customized location for your offline maildir mail repository (this way, we don’t interfere with dovecot’s system-wide configuration file). The contents of the file should be:

# Show more verbose process titles (in ps). Currently shows user name and
# IP address. Useful for seeing who are actually using the IMAP processes
# (eg. shared mailboxes or if same uid is used for multiple accounts).
verbose_proctitle = yes
# Following points Dovecot's IMAP daemon to an offlineimap-created directory (in maildir format):
mail_location = maildir:~/offlinemail/:LAYOUT=fs:INBOX=~/offlinemail/INBOX
Alpine
If you don't yet have a .pinerc file yet, you can safely run alpine once from the command line to create it. In any case, ensure that it contains the following lines (ie, edit the file and find the parts left of each equals sign below, and modify those lines):
# Call local IMAP daemon to get mail.
# Use customized settings when calling local imap, and
#send unnecessary output messages to a file, rather than to Alpine.
rsh-command=/usr/lib/dovecot/imap -c ~/.auto-dovecot.conf 2>> ~/imap.log
inbox-path={localhost/user=NIXUSER}
postponed-folder=drafts
# The following has two entries. The second one is optional; it gives you the
# option to see what the performance is like when you're online of accessing
# GMail's repository directly.
folder-collections="Local dovecot from offlineimap to GMail" {localhost/notls}[],
	Gmail {imap.gmail.com/ssl/user=GMAILUSERNAME@gmail.com}[]
# You may prefer to use a different outgoing (SMTP) server. This choice is
# entirely separate from your choice about where to receive your mail from.
# If you've got a GMail account, here's one option:
smtp-server=smtp.gmail.com:587/tls
There are numerous other settings that may be desirable in Alpine. You can set them through Alpine's menus or by editing .pinerc. You can accept the "remember password" option for your SMTP (outgoing) password when you send your first email from Alpine.
Offlineimap
Next, create and edit the file .offlineimaprc so that it reads:
[general]
accounts = Gmail
maxsyncaccounts = 1
[Account Gmail]
localrepository = Local
remoterepository = Remote
[Repository Local]
type = Maildir
localfolders = ~/offlinemail/
[Repository Remote]
type = Gmail
remoteuser = GMAILUSERNAME@gmail.com
remotepass = my_clear_text_password_Gmail
realdelete = no
nametrans = lambda folder: re.sub('.*Spam$', 'spam',
re.sub('.*Drafts$', 'drafts',
re.sub('.*Sent Mail$', 'sent',
re.sub('.*Starred$', 'flagged',
re.sub('.*Trash$', 'trash',
re.sub('.*All Mail$', 'archive', folder))))))
# You might use the following for your first run...
#folderfilter = lambda foldername: foldername in ['INBOX']
folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail']
You can now run offlineimap from the command line, or set it up to run regularly (method copied/updated below from my earlier post). And after it has synced the first time, running alpine should give you convenient access to your mail, whether you're on or off-line.
More details on running offlineimap using task scheduler:
I've not been able to get the "quick" mode of offlineimap to run reliably, so instead of calling offlineimap every few minutes, I call the following script every few minutes (ie set it to execute regularly using gnome-schedule):
#!/bin/bash
curl -u MyName:MyNamePassword --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if //; print "$2\n" if /(.*)/;' > /home/NIXUSER/tmp/newGmail
touch /home/NIXUSER/tmp/newGmail
touch /home/NIXUSER/tmp/oldGmail
cmp -s /home/NIXUSER/tmp/oldGmail /home/NIXUSER/tmp/newGmail > /dev/null
if [ $? -eq 1 ]; then
 offlineimap  &> /home/NIXUSER/tmp/offlineimap.log
else
 echo is not different >> /home/NIXUSER/tmp/offlineimap.log
fi
cp /home/NIXUSER/tmp/newGmail /home/NIXUSER/tmp/oldGmail
That is, this does a quick check with Gmail to see whether anything is new, and only then does it call offlineimap.
With the same offlineimap / Dovecot setup, you can now easily set up alternative mail clients (Mutt, Emacs-based ones, etc, as well as all the GUIs) to use the same repository.
Next steps
It has for several years baffled me that what you might call non-GUI email clients are so much harder to set up and use than GUI clients like Thunderbird, Evolution, and the Mac and Windows stuff.  I hope GNU/Linux distributions can begin to offer a series of wizards or pre-set packages that make switching between email clients, and setting up standard external connections such as GMail, to be just a click away for non-GUI clients as well as for GUIs.
The present effort (with Asheesh Laroia's expertise) is intended as the genesis of an "alpine-maildir" package in Debian / Ubuntu / etc. Feedback here is appreciated to aid this effort.