Check your IMAP server compliance

The development of James 3.0 is reaching the end, and this new release will bring new protocols support such as IMAP.
One important remaining step is to check the validity of our IMAP implementation:

• are we handling well the requests?
• are we responding well?
• do we respect the concurrent accesses?

As we are implementing tests, we already had an idea about the two first points: we have done what is expected by the RFCs.

But checking for concurrent accesses is a much more complex work, that’s why we decided to use ImapTest.

ImapTest is written by the Dovecot community and so they implemented it to solve the same testing problem.

Installation

In order to install ImapTest you have to compile Dovecot & ImapTest.
I couldn’t find any packages for ImapTest and it needs some libraries from Dovecot, that’s why you have to compile both.

Depending on what is already installed on your machine, you may need to install additional packages, the compilation process will tell you what to install.

Validating IMAP commands

The first point is to have a bunch of messages which will be used for tests, for example by the APPEND command.

Here’s a link to a 10MB file containing messages: http://www.dovecot.org/tmp/dovecot-crlf

ImapTest actually supports common IMAP commands, you can find the list here.

Here’s the command I’m using for validating one command:

$ ./src/imaptest host=127.0.0.1 port=143 user=user1@james.org pass=secret mbox=dovecot-crlf secs=240 -select=0 list=100 logout=0

and its explanation:

• I launch the test on a local James
• I’m using a single user for this validation
• I’m giving a bunch of messages
• This test is payed for 240 seconds (will be infinite if not given)
• The expression is a structured test select=0 list=100,0 logout=0:

1. the select is executed once
2. the list is executed with a probability of 100% (on each iteration)
3. then a logout is executed

Here is the final result:
Totals:
Logi List
100% 100%
10 480660

Validating the whole server IMAP support

You have also the choice to play a script of commands.

This is the chosen solution by the Dovecot team in order to compare different servers by providing a list of scripts.

Here’s the command for that:

$ ./src/imaptest host=127.0.0.1 port=143 user=user1@james.org pass=secret mbox=dovecot-crlf test=src/tests

You just have to add the test parameter, its value should be the path to the script.

Here are the results for spring-jpa James mailbox implementation:
35 test groups: 2 failed, 16 skipped due to missing capabilities
base protocol: 3/366 individual commands failed
extensions: 0/0 individual commands failed

• the skipped test are due to missing thread capability
• the few failing tests are due to missing untagged replies

Conclusion

It was really fun to use a tool provided by an other open source mail server.

It was a great pleasure to see that James IMAP implementation has good results, which is a great point for the incoming release.

We are excited and are willing to reach the perfection, stay connected, we will fix the remaining problems soon.