Hash19: A JSON aggregation library

A Ruby gem to map complex JSON to objects


Assume we are writing an aggregation API that talks to multiple backend systems to mash up what is needed and to push it to a client in a presentable form. To illustrate the need more clearly, let’s build a mobile app called MovieOMeter that suggests what movies to watch in theatres based on the following criteria:

  • vicinity of theatres to the user and movies running in them
  • the movie ratings from multiple sources like IMDB, RottenTomatoes
  • sentiment index of the movie in social platforms like Facebook, Twitter

Each of the services like FB, Twitter, IMDB, TheatreLookup have one or more APIs to fetch these details and for our use case, let’s assume that MovieOMeter features a single page that shows the movie details along with theatres screening the movies and also succinct ratings and sentiment meters based on social activity. Instead of making the mobile app directly talk to different services, the better thing to do would be writing our own aggregator backend that talks to the different services and mashes up the data in a form palatable to the app.

When I made an attempt at writing the aggregation layer, assuming all the services followed JSON over REST protocol, when mashing up multiple JSON structures and transforming it to a structure acceptable to the consumer, I ended up writing lot of boiler plate code.

Hash19 is an attempt at offering a DSL to tame JSON manipulation and help in dealing with common use-cases like

1. Whitelisting attributes

Because not all the data returned by an API is relevant to the context. For example, check out the payload of the twitter API to search for a term. We might only need the most relevant attributes to be exposed to the mobile app.

2. Attribute aliasing and keying

We might want to rename keys to a different form than what is presented to us.

3. has_one and has_many associations

A movie might have many actors, many ratings, many shows (theatre timings and location). So you get the point… Modelling these relationships as associations offers benefits in terms of slicing/dicing, filtering and coding business logic.

4. Lazy loading associations

You get it when you need it! All associations mentioned above might be powered by different services and depending on the use case, a user might or might not be interested in show timings before knowing about a movie. So, it’s better to lazy load the associations and postpone as much as possible the triggers to retrieve the details.

5. Mass injection of associations using bulk APIs

Most REST services offer two version of an API. One that takes in an identifier to return a resource and another version that takes in a list of identifiers and returns a list of resources. There are bulk APIs that return records of the high order along with support for sorting and pagination. It’s useful to automate extracting all movie names returned by a master service and then pass the movie names as params to IMDB, Twitter to get the ratings and trends all at once with a single call.

Enough of discussing the need, show some code is what you ask. Let’s look at a gist that illustrates most aspects of the gem:

Some points to note:

  • A ruby class can be enhanced to be Hash19 friendly by including the Hash19 module
  • MovieOMeter is the container class that will aggregate all information needed by the first page of the app. The contains directive denotes the composing class.
  • The inject directives will be fired after the baseline hash is constructed from a parent JSON. The at param takes a json-path expression to pin point where to extract the identifiers specified by using and inject the data that is fetched by the trigger. The target key can be specified using as param.
  • The whitelisting is done using the attributes directive that takes a list of json keys. If a key has an alias, it can be specified.
  • has_many and has_one associations will be constructed inline if present in the parent JSON or constructed using the trigger that is mentioned as part of the directive.

A lot more details along with examples are available with the source code.