Deliver New xkcd Comics to Your Phone Daily with Node.js, Standard Library Tasks, and Code.xyz
Last week at Standard Library we added the ability to scheduled API executions as tasks from Code.xyz, our online editor for APIs. Scheduled tasks are APIs you can build that run on schedules as granular as per minute, or as long as per week. Whether you want to scrape a webpage for changes, send reminders via SMS or Slack, or create database backups, it’s never been easier to schedule any and all workflow automation tasks.
If you’re not familiar with Standard Library, we’re a platform that enables anybody to turn simple JavaScript functions into scalable APIs in seconds. With the help of our online editor, Code.xyz, you can you can ship APIs to Standard Library right from your browser. To show how easy it is to build APIs and schedule them as tasks, we are going to set up a service that sends the latest xkcd comics to your phone as soon as they’re released. Let’s get started!
Step 1: Create a Function to Scrape xkcd and Send Messages
First things first, we need to create an API that can scrape xkcd, and send us the latest comic via MMS. You can visit the following URL to automatically load the template we have created for this tutorial:
Or, if you already have Code.xyz open, you can click on the “Community API Sources” tab from the landing page and search for “steve/xkcd”.
After opening the template, you’ll see one API endpoint— shown in its entirety below.
The function takes one parameter: a phone number. First, it fetches the xkcd homepage with axios and then uses cheerio to parse the resulting HTML. Perhaps the most unintuitive part of the code is extracting the image source with cheerio, on line 12. It uses a query selector, #comic > img, that tells cheerio to look for a div with the id comic and then find a child of that div that is an img element. That will give you a link like //imgs.xkcd.com/comics/incoming_calls_2x.png. https: is then prepended to the url, so it becomes https://imgs.xkcd.com/comics/incoming_calls_2x.png. That link is passed to utils.mms via the mediaUrl param, which attaches it to the text message.
You can try this out right in the browser. On the right side of the screen, you should see a parameters window. Enter your phone number (include the country code) and click run in the bottom right corner of the screen. You’ll be prompted to log in or signup if you haven’t already and don’t worry, it’s free! After a moment you should get a text and see the following in the results panel, accompanied by a text message to your phone:
You may notice an additional message that says “Sent using Standard Library”. That is added because you are calling utils.mms unauthenticated. To authenticate your request, you need to add a library token to your env.json. Open that file and click in between the pair of quotes next to STDLIB_LIBRARY_TOKEN under the dev key. Press cmd/ctrl k and pick a library token from the prompt. Execute the function again, and you should get just the comic.
Step 2: Schedule Your Function Execution
Now with the function ready, its time to schedule its execution. In the top-left corner of the screen, you should see a tab labeled “Tasks”. Click that, and you’ll see an interface similar to below.
On the left-hand side you can see any currently active scheduled tasks, should you have any. On the right, you can pick an API you want to create a new task for. Pick the API you just deployed — and you’ll move on to the next screen.
There are a few fields to fill in. The Task Name can be whatever you like; it will be used in the left panel as the label. Add your phone number (don’t forget the country code, for example, 15558579305). Now for the schedule, you can see it was inputted as 0 17 * * Mon,Wed,Fri. That is a cron expression that says to execute the function Monday, Wednesday, and Friday at 10:00am PTD, which is around when new xkcds are released. It’s probably worth taking a little detour to figure out how to create a cron expression like that. If you’re familiar with cron, feel free to skip this part.
Step 2b: A (Brief and Incomplete) Introduction to Cron Syntax
Cron is a job scheduler known for its flexibly. But with that power to keep complex schedules, comes some occasionally awkward syntax. A cron job consists of two parts, the schedule and the command. In our case, the command will be the API we want to call, i.e., the xkcd scraper, so we’ll ignore that part.
The schedule consists of five fields, minute, hour, day of the month, month, and day of the week. At each minute, cron checks if the fields match the current time. If they match, the command is executed, otherwise nothing. The five fields and their range of possible values are shown below:
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 7) (0 and 7 are Sunday)
* * * * *For month and day of the week you can use the three letter abbreviations: Sun-Sat and Jan-Dec
In addition to using a fixed value, you can set a field to * which will match the whole range of values that a field can take. Now for a few examples:
0 12 * * * Run at 12:00pm every day
30 13 * * * Run at 1:30pm every day
* 12 * * Fri Run every minute from 12:00pm-12:59pm on Fridays
59 23 31 Dec * Run at 11:59pm on Dec 31There are also a few modifiers which really expand the capabilities of cron:
-is used to match ranges.0–29 * * * *runs every minute for the first 30 minutes of each hour.,is used to match multiple values.* * * * Mon,Wed,Friruns every minute on Monday, Wednesday and Friday./is used to create intervals.*/5 * * * *will run every five minutes. You can combine/with—as well.0-30/5 * * * *will run every five minutes for the first 30 minutes of each hour.
Now, with all of that information, let’s schedule the xkcd scraper. A new xkcd comes out every Monday, Wednesday, and Friday at about 12:00pm UTC, so anytime after that will be good. In the above example it was set to 5:00pm UTC (10:00am PDT). In cron, that means setting the first two fields to 0 17. The next two fields, day of the month and month, are irrelevant to this task, so leave those as *. That gives us 0 17 * *. Now for the Monday, Wednesday, Friday bit, we can use the , operator and set the last field to Mon, Wed,Fri. Putting it all together, we get 0 17 * * Mon,Wed,Fri!
Now, whether you’re a cron wizard, copied the example, or actually read the above— you should have a cron expression ready. Click on the Advanced (cron) button next to the schedule field in the task creation widget and enter it the expression. Press create, and your new task should pop up on the left!
Step 3: Secure Your Endpoint
The discerning security expert in you may have noticed that our API is open to the world. Anyone could call it, with any phone number. Thankfully, there is a quick fix for that. With your API open in Code.xyz, click the purple “Share” button in the bottom-right corner. A modal with pop up, click “Manage Access”, and you’ll switch to the following screen.
This screen describes the permissions around your API. Currently, anyone can call your API with no problem. To change that click the trash bin icons next to the two rows under “Public”. At the next screen click “Revoke Access” and just like that, your API is secured.
Thanks!
And that’s it, thanks for reading! Hopefully you were able to learn a little bit about cron jobs, web scraping, and Standard Library. This tutorial is just one of the many ways you can get started with Standard Library and Code.xyz. If you have feedback or a neat idea you’d like to share, reach out to me directly by e-mail: steve@stdlib.com, or follow me and the Standard Library team on Twitter @StdLibHQ.
Steve Meyer is a recent graduate of Oberlin College and Software Engineer at Standard Library. When he’s not programming, you can find him baking bread, or playing Spider-Man.
