Inbound Emails?
There’s a gem for that
(sharing notes & thoughts from handling inbound emails to Mammoth boards.)
Background
Mammoth now has a really cool feature. In addition to myriad existing ways, you can now post content to a board by just sending it an email. Got an interesting blog post on SaaS sales from Tomasz Tunguz? An article from James Altucher? A new way to run lab experiments? Just forward the email to the board and keep the relevant people upto date.
The Plot
Generally speaking, in many instances, it’s great to be able to automatically handle and do something useful with inbound emails.
Email infrastructure services like SendGrid, Cloudmailin, Postmark or Mandrill etc. all have the ability to parse inbound emails and hand it off to webapps via web-hooks. Each of them slightly different (the params, etc.), but the basic plot is the same.
Griddler
A great gem to standardize this interface across services, is Griddler.IO. Griddler supports the above 4 out of the box, but for other services it seems fairly straightforward to write an adapter (see this section of the docs). I like the way it’s designed. Turns out we’re using Mandrill (lucky), so I got to add the readily available mandrill adapter without having to write one myself.
Griddler automatically adds a default /email_processor route to the app (run bundle exec rake routes and it’ll show up). But it’s fairly trivial to override this and send it to your own controller if you must. Griddler docs are fairly comprehensive, so I won’t belabor this.
DNS Config
It’s best to use a dedicated subdomain for inbounds. This, allowing for variations depending on DNS providers, looks something like this.
Of course, to do anything useful with this subdomain, we’ll need to add MX records for it, but to do that we need to jump to our email provider.
Mandrill
Configuring Mandrill (and I would imagine others are similar in this regard) to start receiving emails is relatively easy. This is where our newly minted subdomain goes and, in Mandrill’s case, it looks a little something like this. (Hitting “Test” should result in an “MX: valid”)
Hitting “DNS Settings” reveals the MX records that we need add to DNS config (so it knows where to route these requests). To the left is what that looks like, give or take.
The values under “Server” and the attendant priority needs to back into our DNS config.
The final step is to configure the webhook in Mandrill, so Mandrill (or whatever email provider) knows where to POST the email to.
Once done, Griddler steps in nicely, parses this email, discards reply-chains, and makes a first class Email object available to our email processor to do whatever we’d like with it. Quickly looking at the code for email.rb and email_parser.rb was really useful to wrap my head around all that’s available.
Attachments
While documentation around most of these are fairly comprehensive across the board, what is not immediately clear is the best way to handle attachments.
The good part is that most email services have file-size limits in the low MB (Mandrill’s is 25MB). The better part is that invoking #attachments on Griddler’s email object returns an ActionDispatch::Http::UploadedFile object straight up. Why’s this awesome? Because we could just turn around and pass it as-is as an argument into Fog’s #put_object method. All the 3rd argument to this method really needs is an object with a #read method that returns contents of the attachment in chunks.
When you distill this down (minus logging and exception handling etc.) it is almost trivially simple. The pseudo code ends up looking a little something like:
Queing Attachment Processing?
I’m just musing here, so nothing concrete. If you have thoughts, or have successfully done this, I’d love to hear more about it.
As best possible, we want to avoid writing files to disk and then processing them, particularly with apps running on ephemeral file systems (Heroku for sure, but also most cloud platforms, right?), apps configured to run on multiple servers behind an LB, or apps that process files in background jobs that run in separate VMs from web processes. Ephemeral file systems don’t survive restarts and are hidden from other VMs, so what one process can see, another may not. I doodled this to help explain... do be kind.
One solution that I want to explore is AWS’ EFS. Since it’s basically an NFS, it would be a persistent and distributed file system that is mounted on all VMs, so all processes (web or otherwise) have a consistent view of all files. And surviving restarts is no longer an issue, since the file systems are not wiped. Have you explored this already? Let me know!
I’m @amitnarayanan on Twitter.
Also, check out Mammoth, a communication and publishing tool I’m currently building.
