De-clutter Your Code with Asynchronous Tasks

One of the big programming trends I’ve been tracking in 2015 is the downfall of the Monolith - a massive, singular application that contains all the code used to run your service or site. Generally the Monolith is discussed in contrast with Microservices, but I’d like to discuss another way you can factor out logic from your core application - repeated, asynchronous tasks.

The traditional approach here is to use crontab to fire off small Unix scripts on a schedule, but I’d like to highlight two relatively new SaaS products that can help ease the burden of setup and maintenance:

• Micorosoft Azure’s WebJobs allow you to run scripts (written in Python, PHP, Java, or a few other languages) on a repeated schedule
• DataFire (our service) allows you to run scripts that call any number of APIs (e.g. MailChimp, MongoDB, GitHub, Heroku) on a repeated schedule

I’d strongly recommend using a third-party service rather than provisioning and maintaining your own task runner. We’re doing this to simplify development, not complicate it!

The Problem

Here’s a (simplified) code flow I highlighted in a previous post:

app.post('/signup', function(req, res) {
  Database.users.add(req.body, function(err, newUser) {
    if (err) return res.status(500).send(err);
      MailChimp.addUser(newUser.email, function(err) {
      if (err) return res.status(500).send(err);
        Salesforce.addLead(newUser, function(err) {
        if (err) return res.status(500).send(err);
        res.send("Success!");
      })
    })
  })
})

What’s wrong with this? Our super-critical /signup endpoint has three possible points of failure! One (adding the user to the database) is surely critical, but the other two (adding the user to MailChimp and Salesforce) can fail safely.

We could improve this code by making it a little more failure tolerant, but regardless we’ll still have these ugly, sub-critical calls to MailChimp and Salesforce attached to our /signup endpoint, failing the Microservices dictum to “do one thing well”.

The Solution

Here’s what our code would ideally look like:

app.post('/signup', function(req, res) {
  Database.users.add(req.body, function(err, newUser) {
    if (err) return res.status(500).send(err);
    else res.send("Success!");
  });
});

Nice and simple! But what about MailChimp and Salesforce?

What we can do is set up a simple, recurring task that monitors our Database for new users, and sends them over to MailChimp, Salesforce, and any other services we might care about. This not only makes the core codebase cleaner, faster, and safer; it also allows us to utilize new services without touching our production code!

Here’s what that might look like on DataFire.

First we get our newest users:

GET datafire.io/services/database/mongodb/documents
function request() {
  var now = new Date();
  var oneDayAgo = new Date(now.getTime() - 1000 * 60 * 60 * 24);
  return {
    collection: 'users'
    many: true,
    query: {
      created: {$lt: oneDayAgo},
    }
  }
}

Then for each new user, we send a request to MailChimp’s listSubscribe method:

GET api.mailchimp.com/1.3/?method=listSubscribe
function request(data) {
  return data[0].map(function(user) {
    return {email: user.email, id: 'MyMailChimpList'}
  })
}

Note that the request() function returns an array this time - one request for each new user. We’ll do the same to create leads in Salesforce:

POST salesforce.com/services/data/24.0/sobjects/Lead
function request(data) {
  return data[0].map(function(user) {
    return {
      email: user.email,
      name: user.name.first + ' ' + user.name.last,
    }
  })
}

We can schedule this Dataflow to run at a certain time every day, or every minute if we want to make sure things are always in sync.

Going Forward

The primary goal here is to pare down your core application into exactly the logic it needs to run. Sub-critical or unrelated tasks should be factored out with extreme prejudice, utilizing microservices, Azure WebJobs, and DataFire Dataflows whenever appropriate. The result is a more stable application, a happier user base, and a more productive development team.

If you’re interested in using DataFire’s beta features like MongoDB integration and up-to-the-minute scheduling, send an e-mail to info@datafire.io