Get up and running with APIs in Node.js

Hey! I get asked a lot how my Twitter bots work by all sorts of people. They use the Twitter API. In this tutorial we will start off with some slow background to APIs, then build up to streaming tweets in real-time and classifying images with the Clarifai API. This is a super beginner-friendly tutorial — if you’re a pro, skip straight to the code.


APIs?!

Application Programming Interfaces — but don’t worry about that. They’re basically special URLs (like https://medium.com) that aren’t just sending back web pages to a web browser. You can interact with APIs to send and receive data.

Cool, how?

Here’s an API. http://itsthisforthat.com/api.php?text

This API changes people’s lives by dishing out hot new startup ideas. Open that URL in the browser. Who needs creativity now?

Damn, that text.

When you open web pages in your browser, you’re sending a GET request over HTTP to the web server. We can also send a GET request in other ways through programs. There are also other HTTP methods — like POST to send data.

Curl

There’s a small program called Curl that can send requests. It’s pretty neat.

In terminal (Linux, macOS) or Git Bash (Windows), run the following curl command. It will send a GET request.
Note: $ indicates the start of a command — you don’t need to type that.

$ curl "http://itsthisforthat.com/api.php?text"
So, Basically, It's Like A Reddit for Traveling Abroad.

Sounds like nomadlist.

The text response from the API will be output in the terminal. Now we could use that text — but to get out the “this” and “that” from the text, we would have to parse the natural language string.

JSON

Instead, this API supports JSON responses by changing the value at the end of the URL to json.

$ curl "http://itsthisforthat.com/api.php?json"
{"this":"Match.com","that":"Farmers"}

Match.com for Farmers. Nice.

Now a program can more easily extract the things we’re interested in from the response. Most APIs these days use JSON but there are other formats like XML.

OK that’s cool, but how do you get Twitter data?!

Hold up. We gotta do some prep.

Somehow we need to handle the data we’re getting back from the APIs. We’re gonna need some code.

Node.js

Let’s use Node.js. Download it and install at https://nodejs.org. Before continuing, make sure you can run the following commands:

$ node --help
$ npm --help

You should get back information of how to use the node and npm commands.

Use git bash on windows. Use any terminal on macOS or GNU/Linux.

Modules in NPM — Node package manager

Requests can get complicated. It’s usually best off using code written by someone else. We will install node modules to help us make API calls.

The first thing we need to do is to run $ npm init to create a package.json file to store information about your application like the module dependencies. Answer each question and hit enter. Don’t worry about the questions — they’re not too important for now!

Now you have a package.json file. This is also JSON format just like we saw earlier in the response! It basically lists all modules that your app needs to run. You can then share this to other people so they can download the modules with $ npm install.

We will be using a module called Twit for using the Twitter API, and Clarifai’s Javascript SDK.

We can use $ npm install --save clarifai@v2.0.12

This installs the clarifai module into a node_modules directory. the --save flag adds the Clarifai module into the package.json file.

We can do the same for Twit: $ npm install --save twit@v2.0.0

Cool. We have everything we need for using Twitter and Clarifai.

We’re gonna need authorisation

So Twitter and Clarifai want to limit access a little. They do this by making “Applications” that your code can use. It basically gives them control so they can identify “who” is making requests.

Make a Twitter account at: https://twitter.com/signup
Make a Clarifai account at: https://developer.clarifai.com/signup/
Either can be one you already have.

Make a new Twitter app at https://apps.twitter.com/ — hit “Create new app”.
Make a new Clarifai app at https://developer.clarifai.com/account/applications/ — hit “Create a New Application”.

Keep these applications open for later.

Pretty similar, amirite. No fluke.

Let’s make an app!

Let’s do this! Let’s stream some tweets and classify the images with Clarifai!

First let’s require our libraries.

const Twit = require('twit');
const Clarifai = require('clarifai');

Initialise some things! We want to put in the authorisation tokens for your Twitter and Clarifai applications. We need to generate the Twitter access token and secret on the “Keys and Access tokens” tab.

const twitter = new Twit({
consumer_key: "insert from twitter app",
consumer_secret: "insert from twitter app",
access_token: "insert from twitter app - generate first",
access_token_secret: "insert from twitter app - generate first"
});
const clarifai = new Clarifai.App("your clarifai client id", "your clarifai client secret");

Let’s stream some things of Donald Trump:

const stream = twitter.stream('statuses/filter', { track: "trump" });

Let’s register a function to run *on* a new *tweet*.

stream.on('tweet', tweet => console.log(tweet.text));

Run the code to see tweets!

$ node file.js

It should start streaming donald trump tweets. Woo.

Now let’s use Clarifai.

It’s a little bit of code to request “tags” — classifying the image getting a probability associated with each tag.

const URL = "http://media3.s-nbcnews.com/j/msnbc/components/video/201607/a_ov_rnc_trump_mashup_160722__872382.nbcnews-ux-1080-600.jpg";
clarifai.models.predict(Clarifai.GENERAL_MODEL, URL)
.then(res => console.log(res.data.outputs[0].data.concepts))
.catch(err => console.log(err));

In res.data.outputs[0].data.concepts is an array of concept objects.
Pretty neat, right?

Let’s wrap this in a function.

 function predict(url, cb) {
clarifai.models.predict(Clarifai.GENERAL_MODEL, url)
.then(res => cb(null, res.data.outputs[0].data.concepts))
.catch(err => cb(err));
}

Cool. We can now call that function. We need to pass in a url to use, and a callback function that takes an error and the array of concepts.

predict(
"http://media3.s-nbcnews.com/j/msnbc/components/video/201607/a_ov_rnc_trump_mashup_160722__872382.nbcnews-ux-1080-600.jpg",
(err, concepts) => {
if (err) console.log(err);
console.log(concepts)
}

Woo. Same thing. Except we can use it again.

Let’s classify the tweets we have. It turns out in tweet.entities.media[0].media_url_https lives a URL to the first image in the tweet. If there’s no image we’ll need to return from that function.

tweet.on('tweet', tweet => {
if (!tweet.entities.media) return; // no image!
predict(
tweet.entities.media[0].media_url_https,
(err, concepts) => {
if (err) return console.log(err);
console.log(concepts[0].name + ': ' + tweet.entities.media[0].media_url_https);
}
)
});

We’re now classifying images in tweets. Awesome!

Got stuck? Here’s some working code. It also saves images to disk with some other libraries. https://github.com/mattburman/twitter-image-grouper

I also demoed this live and explained some things: