IMAP RFCs

functional testing

by Matthieu Baechler / mbaechler@linagora.com


IMAP being a text line-oriented protocol, it’s quite easy to write functional tests based on recorded sessions. It’s also a very well-documented protocol with a good base-RFC and a lot of extension RFCs.

We would like to write a comprehensive testsuite that could be able to check standard respect for various IMAP servers.

James MPT

James is an Apache project that aims at creating a very modular and customizable Java Mail Server.

As very smart people worked on this project in the past, it comes with a functional test tool called MPT.

An imap session looks like the following snippet :

* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] test Cyrus IMAP v2.4.17-caldav-beta10-Debian-2.4.17+caldav~beta10-18 server ready
. LOGIN Bob toto
. OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE IDLE] User logged in SESSIONID=<cyrus-227-1432893832-1>
A1 LIST "" "*"
* LIST (\HasChildren) "." INBOX
A1 OK Completed (0.000 secs 3 calls)
A2 SELECT INBOX
* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UIDVALIDITY 1432893822] Ok
* OK [UIDNEXT 1] Ok
* OK [HIGHESTMODSEQ 1] Ok
* OK [URLMECH INTERNAL] Ok
A2 OK [READ-WRITE] Completed

IMAP sessions are composed of commands, like LIST and SELECT. Each command includes a tag, e.g. A1 which will be used in the answer.

The response includes tagged answers to link response lines with the sent command:

A1 OK Completed (0.000 secs 3 calls)
A2 OK [READ-WRITE] Completed

Answers can also be untagged, in this case they are prefixed with a star:

* LIST (\HasChildren) "." INBOX
* 0 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UIDVALIDITY 1432893822] Ok
* OK [UIDNEXT 1] Ok
* OK [HIGHESTMODSEQ 1] Ok
* OK [URLMECH INTERNAL] Ok

Untagged answers can actually be unrelated to the input command. The only sure thing is that once you reached the next tagged answer, you should have received every information you asked for. Theses information could just be interleaved with unrequested lines.

That said, with some simple constructions, you can create a functional test language and it’s exactly what MPT is for.

If you would like to test the previous IMAP exchange, you could write an MPT test like this:

S: \* OK.*
C: \. LOGIN Bob toto
S: \. OK .*
C: A1 LIST "" "*"
S: \* LIST \(\\HasChildren\) "\." INBOX
S: A1 OK .*
C: A2 SELECT INBOX
SUB {
* 0 EXISTS
* 0 RECENT
* FLAGS \(\\Answered \\Flagged \\Draft \\Deleted \\Seen\)
* OK \[PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK \[UIDVALIDITY \d+\] Ok
* OK \[UIDNEXT \d+\] Ok
* OK \[HIGHESTMODSEQ \d+\] Ok
* OK \[URLMECH INTERNAL\] Ok
}
S: A2 OK \[READ-WRITE\] .*

You can see the MPT language features :

  • C: is for client commands, just append a string behind, it’ll be sent to the server as is
  • S: is for expected answers, the whole line is a regular expression, hence the backslashes everywhere
  • SUB {}: is a section with unordered lines, it will check that all lines are matching one server response in any order. Again, each line is a regular expression.

And that’s it: Simple yet powerful !

What is MPT used for ?

Today, MPT is here to test James at the outer scope. If the output is correct for a given input, the server is doing the right thing.

What’s really interesting is the very small distance between RFC’s terms and the test language : any IMAP-speaking person is able to understand and write any test, nothing is James specific.

What else could we do ?

At Linagora, we are long-time Cyrus IMAP users. But like a lot of other Cyrus users, we’d like to add some value inside our IMAP server and scale it at the cloud level.

We started to think about switching to James for some of our customers but we would really like to check that both servers act the same way : same implemented RFCs and same level of protocol compliance.

We think that MPT is exactly what we need : we write some tests and run them on both Cyrus and James.

How we do it

MPT is mostly some java code. To run on a given server, one needs to implement an interface for methods like “start”, “stop”, “add user”, “create mailbox”.

Cyrus IMAP being a C unix daemon, we made use of the usual swiss army knife to solve the problem: Docker.

For every single test, we start a Docker with an admin account configured and MPT runs its scripts to check the results.

What is coming next ?

We think that writing a conformance test suite that runs on (at least) two implementations is really a great thing for James. Not only will we check that everything works as expected, we will also make sure it works like everyone thinks it should.

We are already writing tests based on the IMAP ACL RFCs and we aim at running them on both Cyrus and James. Running them on a given implementation is just a matter of activating the right Maven profile. We’ll try to make existing MPT tests work on Cyrus, too. And finally, any new test will target both, everytime.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.