Why JMAP is the future of emails?

Published in

Linagora Engineering

7 min readJul 7, 2017

Once in a while, I find myself in a situation where I must send an email from a headless Linux server using only command line tools, often in a non-interactive context. There are several options available, each with pros and cons.

curl

If you’re running on a modern Linux distro, chances are that curl is available out of the box. If you’re lucky enough to run at least version 7.20, you have SMTP support built-in and you can send mail using curl.

The syntax should be similar to the below example, though exact options greatly depend on your environment and target SMTP server:

$ curl \
    --mail-from '<your email address>' \
    --mail-rcpt '<your recipient's email address>' \
    --upload-file <a file containing the message body> \
    --ssl \
    -u '<your credentials>' \
    smtps://<your SMTP server>

The major drawback of the curl approach is that you must prepare the email data in the RFC 5322 format, save it to a separate file and use that file with the --upload-file option of curl. This two-step process is error-prone… Nobody’s ever remember the exact syntax of raw email data, right?

On the good side, you’re using curl ! You don’t need any other tool to send an email on the command line, and you probably already know how curl works. You’ll have to understand the SMTP-specific options of curl, but they have a pretty good documentation.

telnet

You can use telnet to connect directly to a SMTP server and run a low-level exchange with the server, including handshaking, logging-in and message submission. This is a fully interactive process and does not fit well with our use-case, though there are workarounds to this (using Expect for instance might be a good option).

Using telnet, however, gives you full control on the process and might be a better option if you’re working with a SMTP server that curl fails to ensure compatibility with (very unlikely, though) or if you like to do things low-level.

A typical telnet exchange will look like this (authentication is omitted for brevity, and in real life you’ll encrypt the exchange using TLS):

$ telnet smtp.my.domain <port>
Trying x.x.x.x...
Connected to smtp.my.domain.
Escape character is '^]'.
220 smtp.my.domain ESMTP Postfix (Debian/GNU)
HELO Client
250 ...
MAIL FROM:me@my.domain
250 2.1.0 Ok
RCPT TO:<you@your.domain>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
From: Me <me@my.domain>
To: You <you@your.domain>
Subject: Hello from CLIHello,I was sent using a telnet session !Regards,
Me
.
250 2.0.0 Ok: queued as 72EFD400A8EC
QUIT
221 2.0.0 Bye
Connection closed by foreign host.

Other options

Other options include using external tools like mutt, sendmail, mail, etc. Every tool has its own documentation and dedicated website, and several tutorials exist on the Internet. They all work roughly the same way:

You setup configuration for the tool, either via a text file or with command-line options
You prepare email data and pipe it to the command

Once again, you have this two-step process of preparing the data then sending it. Let’s face it, there’s no easy and reliable solution to this simple problem…

OH WAIT!!! THERE IS ONE: MEET JMAP

JMAP is a new protocol, currently in the process of becoming an open standard through the IETF. I’ve written about it before, so I won’t explain the core concepts again, nor how we use it at Linagora in our OpenPaaS product. I want to focus on showing a few JMAP examples so that you’re able to feel the power of the protocol and understand the problem it solves. Moreover, the protocol is designed to also handle calendar and contact data, even though I won’t cover these parts in this article.

As a first step, you need to understand the payload format of JMAP requests (remember this is a JSON-based protocol?). A typical JMAP request is:

[                              <-- 1
  [                            <-- 2
    "getMailboxes",            <-- 3
    {},                        <-- 4
    "#0"                       <-- 5
  ]
]

With:

The outer array. A JMAP request is always an array. This is because the protocol supports batch processing, so you’re able to pass multiple requests in a single HTTP call.
The JMAP request, which is also an array. It is composed of a command, some options and a request identifier.
The command name.
The command options. This is a JSON object and is empty in this example. Options vary depending on the command. For instance, the getMailboxes command we’re using here supports 3 options.
A request identifier. When multiple requests are sent, the server will send one response per given request. It will flag responses with the same identifiers, so that it’s easy to reconcile.

JMAP requests are authenticated requests (you pass authentication data on every call). This is due to the stateless nature of the protocol, you have no session on the server. Authentication in JMAP is done through access tokens; before sending requests, a client must get such a token. Ways to get this token are out of scope of this article, and greatly depend on the server you’re targeting. For instance, in OpenPaaS we use JWT but some other servers might use OAuth2 tokens… Let’s suppose you have your token available, and let’s do some curl !

Getting the list of mailboxes

Mailboxes are mail folders. The easiest call you could make is getting the list of mailboxes for the authenticated user:

$ curl \
    -s \
    -H'Authorization: Bearer MyAuthenticationToken' \
    -H'Content-Type: application/json' \
    -XPOST \
    -d '[["getMailboxes", {}, "#0"]]' \
    <your server's JMAP endpoint>[
  [
    "mailboxes",
    {
      "accountId": null,
      "state": null,
      "list": [
        {
          "id": "8b6d9b50-60db-11e7-8fe8-f5f33b289770",
          "name": "INBOX",
          "parentId": null,
          "role": "inbox",
          "sortOrder": 10,
          "mustBeOnlyMailbox": false,
          "mayReadItems": false,
          "mayAddItems": false,
          "mayRemoveItems": false,
          "mayCreateChild": false,
          "mayRename": false,
          "mayDelete": false,
          "totalMessages": 2,
          "unreadMessages": 1,
          "totalThreads": 0,
          "unreadThreads": 0
        }
      ]
    },
    "#0"
  ]
]

In this example, the user has only one mailbox: the INBOX folder. The complete listing of a mailbox’ properties and the getMailboxes request and response details are in the specification, have a look at it if you’re interested in learning more about mailboxes.

Getting the last unread message from the inbox

In order to query actual messages, we use the getMessageList command. An interesting feature demonstrated below is the ability for the server to return more than one response to a single request, because it executed several operations as a result of receiving the request. This is a nice feature of the protocol that minimizes round-trips.

Some parts of the response are omitted for brevity, but here a sample call to fetch your last unread message in the inbox:

$ curl \
    -s \
    -H'Authorization: Bearer MyAuthenticationToken' \
    -H'Content-Type: application/json' \
    -XPOST \
    -d '
[
  [
    "getMessageList",
    {
      "fetchMessages": true,
      "fetchMessageProperties": [
        "id",
        "subject",
        "from",
        "to",
        "cc",
        "date",
        "preview"
      ],
      "sort": [
        "date desc"
      ],
      "position": 0,
      "limit": 1,
      "filter": {
        "isUnread": true,
        "inMailboxes": [
          "8b6d9b50-60db-11e7-8fe8-f5f33b289770"
        ]
      }
    },
    "#0"
  ]
]
' \
  <your server's JMAP endpoint>[
  [
    "messageList",
    {
      ...
      "messageIds": [
        "a4cb6bc0-618c-11e7-94c6-f5f33b289770"
      ]
    },
    "#0"
  ],
  [
    "messages",
    {
      "notFound": [],
      "list": [
        {
          "id": "a4cb6bc0-618c-11e7-94c6-f5f33b289770",
          "from": {
            "name": "David",
            "email": "ddo@domain.com"
          },
          "to": [
            {
              "name": "David",
              "email": "ddo@domain.com"
            }
          ],
          "cc": [],
          "subject": "Hello from command line !",
          "date": "2017-07-05T14:17:13Z",
          "preview": "Hello, I was sent using a command line JMAP client"
        }
      ]
    },
    "#0"
  ]
]

This call demonstrates filtering, ordering, selective fetch of message properties and paging. All in a single example ! You’ll notice that we first receive the response to getMessageList, containing only the identifiers of the messages matching our criteria, then the message details.

The complete listing of a message’ properties and the getMessageList request and response details are in the specification, have a look at it if you’re interested in learning more about message lists and messages.

Last, but not least, sending a message

Our initial problem was to send an email on the command line. Here’s how this can be done with JMAP:

$ curl \
    -s \
    -H'Authorization: Bearer MyAuthenticationToken' \
    -H'Content-Type: application/json' \
    -XPOST \
    -d '
[
  [
    "setMessages",
    {
      "create": {
        "abe18535-0c53-4b61-b0c5-e711e4149974": {
          "isDraft": false,
          "mailboxIds": [
            "8b7dc7f0-60db-11e7-8fe8-f5f33b289770"
          ],
          "from": {
            "name": "David (Command Line)",
            "email": "ddo@domain.com"
          },
          "to": [
            {
              "name": "David",
              "email": "ddo@domain.com"
            }
          ],
          "subject": "Hello from command line !",
          "textBody": "Hello, I was sent using a command line JMAP client"
        }
      }
    },
    "#0"
  ]
]
' \
  <your server's JMAP endpoint>[
  [
    "messagesSet",
    {
      "accountId": null,
      "oldState": null,
      "newState": null,
      "created": {
        "abe18535-0c53-4b61-b0c5-e711e4149974": {
          "id": "650f71c0-6231-11e7-b579-f5f33b289770",
          ...
        }
      },
      "updated": [],
      "destroyed": [],
      "notCreated": {},
      "notUpdated": {},
      "notDestroyed": {}
    },
    "#0"
  ]
]

Some parts of the response are once again omitted for brevity but with the above request, you successfully sent an email with a single curl call!

There is a lot more, including attachments support, HTML message body, etc. Everything is in the specification.

With JMAP, you gain:

A stateless, JSON over HTTP protocol for both message submission (replacing SMTP) and message retrieval (replacing IMAP or POP).
Simple JSON payloads, allowing seamless integrations in lots of programming languages and environments.
curl-friendliness :)
Easy to use in frontend applications: If you are JavaScript addict, you can also implement your own command line interface by using our JMAP client library (https://github.com/linagora/jmap-client)

Isn’t this the future of emails?

If you are interested in trying this stunning protocol and are looking for a server supporting it, we recommend you to try James http://james.apache.org/

Interested in joining us doing awesome code? Contact us on Github, on Twitter or apply to a job offer on our website.