Cloudant and Node.js Made Simple with the cloudant-quickstart (formerly silverlining) Library

A library for new users

Published in

Center for Open Source Data and AI Technologies

5 min readJan 20, 2017

Editor’s note: In February 2018, the silverlining library has been deprecated and replaced with cloudant-quickstart: https://www.npmjs.com/package/cloudant-quickstart

I frequently talk with developers who are using IBM Cloudant or Apache CouchDB™ for the first time, and I’ve found that they most often have a difficult time coming to terms with these concepts:

Design Documents: a feature that is unique to this database family. They are special JSON documents holding index definitions and other data-manipulation magic. Once mastered, design docs can be used to great effect, but to a first-timer they are baffling.
Multi-Version Concurrency Control: MVCC is the mechanism whereby documents are stored in a tree of revisions, allowing databases to sync changes with other copies without data loss.

Usually, developers want to learn the mechanics of database CRUD and basic queries. They don’t want to have to learn about design documents in order to query or aggregate data, nor do they want MVCC getting in the way of storing and updating their data. What would Cloudant or CouchDB look like without these concepts? Or without show & list functions, update handlers, attachments, replication, and other advanced features?

To find out, I wrote the silverlining library that hides away this powerful-but-esoteric functionality and concentrates on making data manipulation simple and intuitive.

“Hi ho silver lining, and away you go …”

Introducing silverlining

The silverlining library is a Node.js module that can be used with Cloudant and allows:

database creation
document insert, update, bulk insert and delete
document get, multi-get and bulk fetch
database querying
aggregation for counting, totalising and statistics

All of this is achieved without users dealing with any of Cloudant’s inner complexities.

Installing

Use the library in your own project like so:

> npm install --save silverlining

Then, use it within your code by adding the URL of your Cloudant instance and defining which Cloudant database you want to work with:

var url = 'https://myusername:mypassword@host.cloudant.com';
var cities = require('silverlining')(url, 'cities');

Creating a database

A new Cloudant database can be created on first-use with the create function:

cities.create();

All the function calls we make here are going to be asynchronous: they return a Promise, which you handle with then and catch functions, but for brevity, I’ve omitted much of this scaffolding from the code samples. For example, to log the output of the info function you could do:

cities.info().then(console.log)

Now, we are going to import some data from geonames.org, which stores the location and population of cities around the world.

Inserting data

Single documents can be added to a database with the insert function:

var city = { 
  _id: '3041563', 
  name:'Andorra la Vella', 
  latitude: 42.50779, 
  longitude: 1.52109, 
  country: 'AD', 
  population: 15853, 
  timezone: 'Europe/Andorra' 
};
cities.insert(city);

If an _id property is supplied, then this field becomes the key field of the JSON document stored in Cloudant. If omitted, an '_id' is automatically generated by the database.

The same function can be used to insert multiple documents — simply pass in an array of objects instead:

var multi = [
  {
    _id: '1000501',
    name: 'Grahamstown',
    latitude: -33.30422,
    longitude: 26.53276,
    country: 'ZA',
    population: 91548,
    timezone: 'Africa/Johannesburg'
  },
  {
    _id: '1000543',
    name: 'Graaff-Reinet',
    latitude: -32.25215,
    longitude: 24.53075,
    country: 'ZA',
    population: 62896,
    timezone: 'Africa/Johannesburg'
  }
];
cities.insert(multi);

The array of data can be as long as you like. Let’s get some data from Github and store it in a local file:

curl https://raw.githubusercontent.com/glynnbird/cities/master/cities.json > cities.json

It can then be imported in batches with one function call:

var mydata = require('./cities.json');
cities.insert(mydata);

Records can be retrieved by their IDs with the get function singly:

cities.get('1000543');

Or get records in batches by providing an array of IDs:

cities.get(['1000501', '1000543']);

Individual records can be updated by supplying the ID and a new document body to the update function (or just a new object):

cities.update('1000501', newdoc);

A document can be deleted with the del function:

cities.del('1000501');

Assuming we have managed to create a suitable data set, then we next need to query it.

Querying data

The database can be queried by any field using the query function, passing in a query object:

cities.query({ name: 'York' }).then(console.log);
[ { _id: '2633352',
    name: 'York',
    latitude: 53.95763,
    longitude: -1.08271,
    country: 'GB',
    population: 144202,
    timezone: 'Europe/London' },
  { _id: '4562407',
    name: 'York',
    latitude: 39.9626,
    longitude: -76.72774,
    country: 'US',
    population: 43718,
    timezone: 'America/New_York' } ]

The query can also contain Cloudant Query selector operators:

// find cities with a population > 5m
cities.query({ population: { '$gt': 5000000} })// cities with population > 5m in Brazil or USA
cities.query({ population: { '$gt': 5000000}, country: { '$in': ['BR','US']} })

Aggregation

Create aggregated views of your data with the count, sum and stats functions.

The count function simply counts things:

cities.count().then(console.log)
23515

It can also count things grouped by other fields within the document:

// get counts of cities by country code
cities.count('country').then(console.log);
{ AD: 2,
  AE: 13,
  AF: 48,
  AG: 1,
  AI: 1,...

The sum function calculates totals:

// get sum of all population fields
cities.sum('population').then(console.log);
2694222973

You can also use sum to group the totals by other fields:

// get totals of population grouped by country
cities.sum('population','country').then(console.log);
{ AD: 36283,
  AE: 3272938,
  AF: 6308267,
  AG: 24226,...

The stats function calculates statistics on one or more fields, which can be grouped by other fields:

// get stats on cities' population by timezone
cities.stats('population', 'timezone').then(console.log);
{ 'Africa/Abidjan': 
   { sum: 8399817,
     count: 57,
     min: 15068,
     max: 3677115,
     mean: 147365.2105263158,
     variance: 240877692698.44693,
     stddev: 490792.9224208993 },
  'Africa/Accra': 
   { sum: 7064394,
     count: 58,
     min: 18077,
     max: 1963264,
     mean: 121799.89655172414,
     variance: 96426027195.8169,
     stddev: 310525.4050731065 },...

How does this work?

It’s still Cloudant under-the-hood, but this library is taking care of a few details on your behalf:

When you create a database, a Cloudant Query index is created that indexes all fields. This index then powers the query operator.
The update and delete functions hide revision tokens from you — behind the scenes these operations fetch the latest revision, overwriting the data with your supplied document. If it doesn't succeed initially, it tries again later, up to three times, at increasingly longer time intervals.
The count, sum and stats functions use Cloudant's built-in MapReduce engine, creating the required Design Document if necessary.

In production systems, it’s important to understand design documents, MVCC and the other features of Cloudant, but when you’re getting started it’s helpful to be able to write data quickly and create queries without any fuss.

Is this free?

Yes! This is a free, open-source library that can work with any Cloudant account.

Code: https://github.com/ibm-cds-labs/silverlining
NPM page: https://www.npmjs.com/package/silverlining

Feel free to give it a go, and we’d welcome any feedback through the project’s GitHub Issues page.