Building a Scheduled Newsletter in Ruby with IronWorker and Mailgun

As developers, the applications we build have an amazing array of functionality, and grow more impressive by the day. Amongst the varied functionality sits job processing, and process scheduling — features that crop up in almost every application we build! Whether that be scheduling an email, scheduling a data scraper, or scheduling a payment processor, we’ve had to rely on fairly ancient, and often confusing technology, such as Cron Jobs, to enable this.

Setting up Cron Jobs has long been a thorn-in-the-side of developers, and I’ve been told, also very confusing. Apps like WordPress have built in Cron features, but a simple Googling will sharply reveal the multitude of issues experienced. Happily, in the modern API-driven world, there are task scheduler SaaS applications offering us a much simpler solution.

Instead of having our own servers run Cron jobs, job scheduling and worker APIs offer us the same functionality, with a much more modern feel. In today’s case, we’re going to use IronWorker from Iron to schedule a newsletter with useful tech articles from Hacker News. Iron offer a hosted SaaS solution for task scheduling and job processing. IronWorker is their highly available and scalable task queue / worker service.

Our newsletter scheduler will run at 1:30 am every morning, ensuring the articles are in our email inbox, ready for breakfast! Personally, I like to spend a bit of time each morning reading up on interesting news and articles before I start my day.

To build our RSS reading application that aggregates the tech articles, pushes them into email format and actually sends the email, we will be using Ruby. No need for any fancy frameworks, or servers today — they’re not needed! We will bundle a couple of gems to aid us in parsing and processing, but other than that, our app is a simple vanilla application.

So, without further ado, let’s get on building our dynamic newsletter. To follow along with this tutorial, the complete sample application is on GitHub here.

Build a dynamic newsletter with Ruby

Realistically, the main focus of this article is setting up a scheduled worker in Iron using the new Docker workflow, and with bundled Ruby dependencies (something that cost me a lot of trial & error time). With that being the case, our “dynamic newsletter” is going to be super simple, but it does run through the Iron/Ruby workflow from start to finish.

Our newsletter needs to take a feed of technical articles that we will find interesting. For this, I’ve chosen HN and their RSS feed. The Hacker News feed is in a standard XML format, and as we don’t have time (or the need) to write an XML parser ourselves right now, we’ll find a Gem and use that.

I’m going to work on the assumption you’ve got Ruby installed on your system, and if not; I’d recommend heading over to Rbenv and finding out how to do so. I’m using Ruby 2.5 for this app. Alongside having Ruby installed, you’re going to need Bundler.

In your current working directory, create a Gemfile and enter in the following:

Here we’re bringing in the official Iron Ruby gem, the Feedjira Gem to parse our RSS feed, and the official Mailgun gem to deal with our email sending. Go ahead and create a file named articles.rb, and add in these dependencies:

Important note

The line require_relative 'bundle/bundler/setup' is very important to us in this project. Without including this line, or by using the standard require statement, our dependencies would not vendor at all within Docker or Iron . (There is some more information on this later on in the article, too.)

We need to write our article feed aggregation so add the following to articles.rb :

Now we have a Class setup for our article aggregation, we will wrap those articles up into an email, and have our application send that email as its runtime function. Create a send_email() class that reflects this:

…And finish the file up by calling:

send_email()

Our send_email class instantiates a Mailgun client instance, and prepares the newsletter email. In the message_params we take the to address from our payload.json file, via the IronWorker gem . By doing this, we could change the email address that the newsletter is being sent to from within Iron.io, and without having to change any external files. The Mailgun API allows us to return a sending result that I’ve included in our application so Iron.io will store it in a log for any future reference. (If needed).. (Which in this case, it won’t be.)

Now, there are just two JSON files left to create to get our application working. Firstly, if you have not already, create your payload.json file, and enter the email address you wish the newsletter email to be sent to, in standard JSON format:

{
  "email": "robin@percy.pw"
}

Now, create the file iron.json and enter your Iron.io credentials into it: (this can also be done through setting environment variables.)

{
  "token": "YOUR_TOKEN_HERE",
  "project_id": "PROJECT_ID"
}

In the real world, or if this article was focused on the actual email itself, I would have spent time formatting the article list into a much more visually pleasing affair. If you would like to expand on this simple example, it may be a good idea to think about dealing with the RSS feed using proper storage, perhaps creating class instances for each article, or even storing the items in a database if you were building an application featuring repeated requests of the article list.

You could even provision an IronCache service in Manifold to store the list in a key/value format, or alternatively, provision a RedisGreen service in Manifold, and use Redis to do the same;

With that, we have our application set up and working. Now we need to build it into a Docker image, and push it up to Iron to be scheduled!

Setting up IronWorker to run our Newsletter app

Iron have taken full advantage of Docker’s increasing popularity, and built the IronWorker dev flow to reflect that. This means that our scheduled worker applications (regardless of language) can be built into Docker images for easy distribution / dependency management. That being said, if you’d prefer not to use Docker, applications can be uploaded via the old workflow: packaged into a zip file and uploaded through the Iron CLI.

For this tutorial, I will be following the newer workflow, so if you are following along, the next thing you need to do is ensure that Docker is indeed installed and running on your machine. From your Terminal, run:

$ docker info

If this doesn’t print information about your Docker installation, head over to these instructions (for Mac OS) and follow them. With Docker installed, you will also need the Iron CLI app installed. To install is as simple as running the following command:

$ curl -sSL https://cli.iron.io/install | sh

You can ensure Iron CLI was installed correctly by running:

$ iron --version

With both installed, you’re now ready to package up your code, and send it onto Iron to be scheduled. Being a Ruby app, the Gem dependencies need to be bundled into your Docker image before being run on IronWorker. Now, the process for this is fairly straight-forward, yet the instructions seem to be different on every page they’re printed on. This can lead to major confusion and actually meant I was forced to spend multiple hours trying to figure out exactly what I was doing wrong! Following the process in this article will save you the confusion that I had.

To vendor in the required dependencies, run the following:

$ docker run --rm -v "$PWD":/worker -w /worker iron/ruby:dev bundle install --standalone --clean

A few things to note from this command: Firstly, the iron/ruby stack container image is set to use the ruby:dev version. This just ensures we always have an updated Ruby running. Secondly, the gems being bundled are tagged with --standalone and --clean . This means our bundle can work without the Bundler runtime, and any unused Gems are uninstalled. Please ensure you run this command anytime you do add any dependent gems into your application.

As I mentioned earlier, you may have noticed that in articles.rb , instead of there being the standard require command on the first line, we include the following so it uses the vendored gems :

require_relative 'bundle/bundler/setup'

To test the application locally, within the Docker container, run the following:

$ docker run --rm -it -e "PAYLOAD_FILE=payload.json" -v "$PWD":/worker -w /worker iron/ruby ruby articles.rb

With the application running successfully in our local Docker container/image, we know that we can package it and upload it, safe in the knowledge that it will work when run as a scheduled task by Iron. Knowing that our application does indeed work as intended, we can package it up and push it up to the Docker Hub. Before doing so, we’re gonna need a Dockerfile in our code. If you haven’t already done so, please register an account on the Docker Hub website.

Create your Dockerfile and add in the following:

FROM iron/ruby
WORKDIR /app
ADD . /app
ENTRYPOINT ["ruby", "articles.rb"]

The Dockerfile tells our runtime that our container/image is based off the official iron/ruby base. The ENTRYPOINT is the command used by the package to exec our desired ruby file. With the Dockerfile now included, you can package up your application in a Docker image ready to send over to IronWorker to schedule up.

$ docker build -t USERNAME/newsletter:0.0.1 .

Naturally, when running this command, you must swap in your Docker Hub username and the package name. The 0.0.1 is the package version, which you can update whenever you make and deploy changes to your code. Once you’ve built that code into the image, it’s worth testing the image one last time before deploying, just to ensure it has been packaged correctly. We can test the image using a very similar command to our local test statement earlier, that looks as follows:

$ docker run --rm -it -e "PAYLOAD_FILE=payload.json" USERNAME/newsletter:0.0.1

Assuming that once again the command runs our code as expected, the test has proven successful, and you may now push the image up to the Docker Hub. To upload the image onto the Docker Hub registry, run the following:

$ docker push USERNAME/newsletter:0.0.1

Great! We’ve got our newsletter application built into a Docker image, and uploaded onto the Hub ready for use. Now, we need to Register the Docker image with Iron.io, to create a ‘code package’ in their system that can be scheduled to run in tasks. Using our recently installed Iron CLI tool, we can do just that!

$ iron register USERNAME/newsletter:0.0.1

With our code / Docker image registered in Iron, we can now use either the REST API, the CLI tool, or the web admin panel to schedule the code to run whenever we want it to. To perform a quick test to ensure the code runs on Iron.io as it should, you can run the following command:

$ iron worker queue --payload-file payload.json --wait USERNAME/newsletter

Running this will queue a task to execute our code, with the --wait flag keeping the connection in our Terminal open, allowing us to view the completion result. Of course, if you’d prefer to test the scheduling in the Admin panel, that is perfectly fine too:

Scheduling our Nightly Newsletter

Log into the Iron Admin HUD , and navigate into the Worker section of the project you created for this application. In the Tasks subsection, you will notice there is a “complete” task already sitting there. This is the task we queued up to test from our Terminal.

To set up the nightly-scheduled task, you can use the Admin HUD in the “Scheduled Tasks” tab, which is all rather self-explanatory. You can also schedule tasks to run programmatically, which in the real world, is probably more likely. Scheduling a task can be done with a POST request to:

https://worker-aws-us-east-1.iron.io/projects/{Project ID}/schedules

The JSON encoded body of your POST request should look similar to the following:

And with that, we have a scheduled nightly Dev newsletter! The scheduled task will appear in the Iron HUD like so:

Conclusion

The main point I’d like to focus on now that we’ve come to the end of this tutorial, is the dependency vendoring / bundling which for a time, gave me quite a lot of trouble. The main point of trouble, in hindsight, is the fact that most of the existing tutorials don’t fully cover topics, leaving questions unanswered. Initially, I had a lot of trouble bundling the gems required for this application and running it once pushed to Iron, but happily, now that I’ve figured it out, I can share the knowledge with you!

Although this article covers a rather small app example, you can use this app as a base to continue building, and move onto something amazing. In a real-world situation, IronWorker would be used to send huge amounts of emails or notifications in scheduled batches.

A common pattern, for example, is to send a nightly report to customers. Say you have 1M customers. Generating this report might access your database, make use of 3–4 different APIs, and then finish by sending an email using SendGrid, Mailgun, and Mandrill. If each customer took 10 seconds to process, serial execution would take 115 days to send this report. Obviously, not something that can be done without some element of scaling. Enter IronWorker.