Node App + Slack Plugin: Staying Up to Date with COVID-19 Data
In this article, we’ll look at how to make a simple Node.js app to scrape COVID-19 data from a reliable source, build a Slack message with the data and then send the message at a scheduled time each day using a webhook.
Part 1: Data, Data Everywhere
We’ll obtain the COVID-19 cases data from WorldOMeter (a well-maintained site, vetted by the American Library Association), because we want reliable numbers, not misinformation. To scrape the web page for the relevant stats, we’ll use the Cheerio module (assuming we already have node installed). And we’ll store the stats in the reader’s preferred DBMS (I used Postgres and the Sequelize ORM to set up a connection to my local server and model the table).
We’ll be scraping and storing a few data points: the total number of COVID-19 cases, new cases, total deaths, new deaths and total recovered.
Let’s set up our table, ‘livestats’, for that.
The findByCountry function will help us query the table and find the stats by country, while findCasesByCountry will bring up the number of cases. These are class methods we are defining to add functionality to our models for when we need information by country.
Now that we have our models set up, let’s inspect the HTML elements of the DOM from the website to plan how we’ll scrape the relevant data. We’ll first need a jQuery Selector to fetch each row of the table (one row per country).
$(‘#main_table_countries_today > tbody:nth-child(2) > tr’)
The selector will return an object of the selected DOM elements that we can iterate over to get the set of data for each row. We are interested in the children property of the HTML element, and more specifically, the odd children (1 through 11 in terms of indices).
For example, to get to the first country’s name from the table, the selector will look something like this:
$(‘#main_table_countries_today > tbody:nth-child(2) > tr:nth-child(2) > td:nth-child(1)’)
For each row, we’ll store the data in the casesByCountry object, create an instance and add it to our Sequelize model.
Before loading the HTML page into Cheerio, we’ll need to make an HTTP request to our website URL. We’ll use Axios to do this. Add in some routine string manipulation to polish the data, and we have our scraping algorithm.
Part 2: Load up the ship
The Slack messaging app will serve as our front end.
To start, we’ll fetch the data stored in our database and build the payload. Then, we’ll send it using Slack’s postMessage API. (There are some prerequisites here: a configured Slack app and a webhook that we’ve generated to a channel in the workspace.)
First, let’s build the graph portion of the message.
If our front end was a plain website, then we’d have the luxury of including a script tag to our HTML to render a chart. But the best we can do with the Slack API is to download a chart, host the image and attach the link to our payload.
To begin, we’ll fetch the number of cases from our database for the country we’re interested in, using our predefined findCasesByCountry class method. (An array of model instances will be returned, which can be iterated over using the ‘get’ function.)
Then we’ll feed this data into a dynamic graph generator (I created an HTML Canvas Element using Chart.js and added the .toDataURL() method to export the graph to a PNG file, but I’ll leave this part to the imagination of the reader.) Once we have the image, we’ll host it on an image hosting website.
By the end of this step, by whatever means the reader deems most convenient, we’ll have a URL to our graph image.
Now, let’s build the actual payload that we’ll pass to the postMessage API.
We’ll use our predefined findByCountry function to get all the statistics for our country of choice. Then we’ll populate the payload using template literals to enclose dynamic data.
Part 3: Lift the anchor
We are almost done. The last task is to set up a job that will periodically scrape the data, sync our model with the database and post our previously built dynamic message using node’s http-service module. We’ll use Node Schedule to schedule the job.
Part 4: The ship has sailed
In this article, we went over how we can scrape the WorldOMeter website to obtain timely data on the COVID-19 pandemic (using Cheerio.js) and set up a database to store that data (Postgres and Sequelize). Then we looked at how to feed that data into a simple line graph and a dynamic payload and send the payload to a Slack channel via a webhook (Slack API and Http-service) at a time scheduled by a scheduler (Node Scheduler).
We hope this was helpful!
P.S : We’re hiring at Heady! 👋 If you would like to come work with us on some really cool apps please check out our careers page.
