Analysing Cloudant JSON in RunKit JavaScript Notebooks
Using the simplified cloudant-quickstart (formerly silverlining) npm module
Computational notebooks provide a workspace where programmers and data scientists can process, visualise and analyse data on a web page. Usually, notebook code is written in Python, Java or Scala — not necessarily my languages of choice.
As a JavaScript programmer, I haven’t really used notebooks.
RunKit: bringing notebooks to JavaScript
RunKit is a notebook service that uses JavaScript as its language of choice, and it allows you to ‘require in’ any Node.js (npm) package and use that in your code. This playground environment makes it really useful to process data, document your approach, and share it with others.
RunKit is useful for analysing and sharing data that is defined in the notebook. Here’s a simple example analysing US elections margins. The notebook service really comes to life, however, when it’s using data from a database. I’m going to show how to pull data from a Cloudant database using the silverlining package — a very simple Node.js library that allows data to be created, queried and aggregated from the Cloudant NoSQL service.
Creating some data
Sign up for a Cloudant account, and in the Cloudant dashboard click Create Databases and name it cities.
In the Replication tab, create a new replication job from https://reader.cloudant.com/cities to your local cities database. This will copy thousands of JSON documents to your Cloudant account.
Note: This replication will likely cost you a few cents worth of Cloudant usage. But as long as your bill remains under $50 for the month, it’s free.
Next we’re going to analyse the data in a notebook.
Connecting to the database
Create a new RunKit notebook. In the first code block, you could write code like this:
var url = 'https://myusername:mypassword@myhost.cloudant.com/mydb';
var db = require('silverlining')(url);Editor’s note: In February 2018, the
silverlininglibrary has been deprecated and replaced withcloudant-quickstart: https://www.npmjs.com/package/cloudant-quickstart
But this would be a mistake. All RunKit notebooks are public, so you would be revealing your Cloudant credentials to the world. Best practice is to visit the environment configuration page, create an environment variable (for example, URL) that contains your full Cloudant URL, including the username and password, and then reference it in your code (for example, process.env.URL):
var db = require('silverlining')(process.env.URL + '/cities');This way, your notebook still works, but your secrets are hidden.
Querying data
You can query your database using the query function. In this case, you’ll get all the cities whose population is greater than 5 million:
// create Cloudant connection
var db = require('silverlining')(process.env.CLOUDANTURL + '/cities');// query cities with population greater than 5 million
var cities = await db.query({'population': { '$gt': 5000000 }});
Notice how the call to db.query is preceded by the keyword await. This keyword is a new JavaScript feature that allows asynchronous function calls to be created as synchronous calls. It greatly simplifies step-by-step code, as used in notebooks like RunKit. Normally, an await statement can only be used in an async function, but RunKit lets you get away with that detail!
Because the returned data contains values for latitude and longitude, RunKit allows you to render the list on a map. In fact, it does so by default:
You could also get only British cities:
// cities in Great Britain
var cities = await db.query({'country': 'GB'});
Or fetch a city by its id individually:
// get London, knowing its id
var london = await db.get('2643743');
Or get ids in batches:
// pull two cities at once
var cities = await db.get(['2643743','2642607']);
At any point, you can switch from the map view to the properties view:
Clone the notebook
Here is my cities notebook on RunKit.
Once you create a RunKit account, you’ll be able to clone the repo and run my example cells for yourself. Just remember to create an environment variable for your Cloudant account. If you call yours CLOUDANTURL and drop the database name from the end of the string, you can run my code as-is.
When you’re set up, hit the green repeat button to the right of the cell to re-run it, or you can hit shift + return as a shortcut.
Mixing with other data
You can also bring in other data sources for your notebooks. In this notebook, you’ll fetch the current position of the International Space Station and combine it with the city data from my previous example.
// get current position of the international space station
var request = require('request');
async function currentPosition() {
var issurl = 'http://api.open-notify.org/iss-now.json';
var obj = JSON.parse(await require('request-promise')(issurl)).iss_position;
obj.latitude = parseFloat(obj.latitude);
obj.longitude = parseFloat(obj.longitude);
return obj;
};// fetch current position
var iss = await currentPosition();
Then you can query your Cloudant cities database to find nearby cities:
// create Cloudant connection
var db = require('silverlining')(process.env.CLOUDANTURL + '/cities');// get cities "near" the ISS (sometimes there are none — there's a lot of empty out there)
var t = 5; // five degrees of tolerance
var cities = await db.query(
{
'latitude': {'$gt': iss.latitude — t, '$lt': iss.latitude + t},
'longitude': {'$gt': iss.longitude — t, '$lt': iss.longitude + t}
}
);
What’s your next JavaScript notebook?
As a JavaScript developer, I hope you’ll get value out of analyzing data in a notebook. I chose to require my npm module, silverlining, because it simplifies database interactions with Cloudant. But remember that in RunKit, you can require in any Node.js package you need.
To wrap up, here are the Notebooks I made for this article:
Share your own RunKit notebooks in the comments, and please consider sharing this article with other Medium readers by clicking the ♡ here.
