Loki’s Way: LokiJS, the idiomatic way

Enter Loki

LokiJS is a fast in-memory datastore written in JavaScript, which prioritises performance and a small footprint over everything else. This introduction is aimed at making the best usage of the library for those who are already familiar with it. You can probably follow it even if you’re not familiar with LokiJS, but for a more basic tutorial and overview of LokiJS feel free to head over to lokijs.org or the github page for the code itself.

If you are very familiar with LokiJS, you may well skip on to the Resultset and Branching section below.

Before moving onto the actual introduction, a quick clarification on why we claim that LokiJS is “fast” (you can read a full article here).

LokiJS data collections maintain an index which allows find operations to achieve extremely fast performance, thanks to the fact that indexes are sorted and search is performed through the fastest search algorithm for sorted arrays, the Binary Search Algorithm.

A second vital feature to speed things up is Loki’s (yes — the Norse god himself) concept of DynamicView. In short, it’s a self-updating subset of the collection data based on filtering and sorting. (Self-updating means that when objects are inserted or modified, a DynamicView will check if that object is to be added or removed from it). There is a hit in performance at creation time, then speed increases to near get() speeds. DynamicViews also maintain internal indexes, to further accelerate speed.

Lastly, a final feature that helps performance is events. Whenever data is added, updated, removed or an error occurs, the collection emits events (LokiJS contains a minimalist custom rolled-out version of EventEmitter). This means you can do further data processing before and after inserts, updates, deletes etc. by adding listeners, and they are executed in an asynchronous fashion so the process will only execute those when there’s time for it.

“Loki’s Way”: DynamicView

So what is Loki’s vernacular?

Let’s move onto an actual example and let’s assume you’re using LokiJS in a node.js environment and you’re building some kind of app dealing with Doctor Who.

First we create a datastore.

var loki = require(‘lokijs’), db = new loki('test.json');

The we create a collection:

var doctors = db.addCollection(‘doctors’);

then we insert some data:

doctors.insert({ name: ‘David Tennant’, doctorNumber: 10 }); doctors.insert({ name: ‘Matt Smith’, doctorNumber: 11 });

So far so good, if we inspect our collection we can also see that LokiJS has added some metadata to the original objects:

> doctors.data
[ { name: ‘David Tennant’,
 doctorNumber: 10,
 objType: ‘doctors’,
 meta: 
 { version: 0,
 created: 1415567438149,
 revision: 0 },
 id: 1 },
 { name: ‘Matt Smith’,
 doctorNumber: 11,
 objType: ‘doctors’,
 meta: 
 { version: 0,
 created: 1415567451598,
 revision: 0 },
 id: 2 } ]

Now, we would like to create a DynamicView that allows us to find doctors whose “doctorNumber” property is greater than 8, and we’re going to call this view “newerDoctors” (that is — the doctors after the series resumed in 2005). This is accomplished by doing:

var view = doctors.addDynamicView(‘newerDoctors”); view.applyWhere(function (obj) { return obj.doctorNumber > 8; }): view.applySimpleSort('doctorNumber', true);

First we created a view, then we passed a filter function (which takes a single database object as an argument and returns a boolean indicating whether the object should or not be part of the view), then we applied a sort specifying the property to use for sorting, and we passed a boolean indicating the sorting is in descending order (this is an optional parameter which defaults to false, only specify it if you want descending results).

So far all records satisfy the view filter so they both turn up in

view.data()

Let’s add two more records:

doctors.insert({
 name: ‘Paul McGann’,
 doctorNumber: 8
});
doctors.insert({
 name: ‘Peter Capaldi’,
 doctorNumber: 12
});

If we inspect the data again we get:

[ { name: ‘Peter Capaldi’,
 doctorNumber: 12,
 objType: ‘doctors’,
 meta: { version: 0 },
 id: 4 },
 { name: ‘Matt Smith’,
 doctorNumber: 11,
 objType: ‘doctors’,
 meta: { version: 0 },
 id: 2 },
 { name: ‘David Tennant’,
 doctorNumber: 10,
 objType: ‘doctors’,
 meta: { version: 0 },
 id: 1 } ]

Our doctor number 12 is showing while doctor number 8 is not. The DynamicView operated that update by itself, evaluated the two documents inserted and correctly excluded doctor number 8 and included number 12.

Now that we are familiar with the concept of DynamicView, let us analyze which LokiJS features are suited to what use-cases.

Advanced: Resultset and branching

Resultset is an in-built class holding a subset of data which is the result of a query. One of the most attractive features of a resultset is that you can coll a branchResultset() method which will allow to apply further/modified filters to a query you already operated without changing the original resultset.

Knowing this let’s see how to make best use of Core LokiJS, Resultset and DynamicView.

We can general sub-divide the use of LokiJS in three types:

(1) Core : Simple app can just use highest performing core functions (get, find, insert, update, delete).

(2) Chaining / Resultset : using resultsets you can save intermediate results, branch off into subqueries, sort, limit/offset, update, mapReduce. Here’s an example, assuming you’re working on some kind of app dealing with a customer base:

// get irish customers
var irishCustomers = customers.chain().find({ ‘country’: ‘ie’ });
// branch just long enough to determine count of customers under 30 without filter affecting irishCustomers results
var lt30count = irishCustomers.copy().find({ ‘age’: { ‘$lt’ : 30 }).data().length;
// branch and retain list of (irish) customers over 30, sorted by number of orders
var gt30users = irishCustomers.copy().find({ ‘age’: { ‘$gte’: 30 }).simplesort(‘orderqty’);
// utilize branching again to limit (without affecting) our gt30users resultset and determine average expenditure of top 10 customers
var top10over30 = gt30users.copy().limit(10).mapReduce(sumOrders, avgExpenditure);

(3) Dynamic Views : allows you to define ‘core’ frequently used filters, similar to a main branch of a tree. These views will be highly optimized, narrowing down results so that you can further branch off of it (if needed) to perform your ‘edge’ query cases. For the above example you might create a view for irishCustomers, over/under 30 users or a combination. These views will be highly optimized. At any point you can call branchResultset() to apply more filters, limit, resort, and mapreduce on subsets.

var io30 = customers.addDynamicView(‘IrishOver30');
io30.applyFind({ ‘country’: ‘ie’ }); 
io30.applyFind(({ ‘age’: { ‘$gte’: 30 }); io30.applySimpleSort(‘orderqty’);
var top10over30 = io30.branchResultset().limit(10) .mapReduce(sumOrders, avgExpenditure);

Conclusion

An idiomatic LokiJS program is one that leverages the ability of the datastore to create “branches” of result sets, that sets up a number of dynamic views for all recurring queries/filters, and simply invokes view.data() to retrieve the up-to-date results, and uses core functions for retrieval/manipulation of individual results.

Note: you can create a dynamic view even when you have no data in your collection yet, so even creation will be fast.

The above example is available as a gist on github.

In the next posts we’ll talk more in detail about collection events, and upcoming features.