Avoiding the Backend Monolith with Seneca.js

In our last post, we talked about our strategy for presenting the Barnes collection online. We talked about TMS (The Museum System) and how it is less than ideal for holding our data. Our goal is to export data from TMS and import it into an Elasticsearch index. We also want to do image processing on the collection to provide a visual browsing interface. And since the data in TMS is not fixed, all of this needs to update every night.

Making all of this happen requires orchestrating a number of tasks:

• Export data from TMS to a CSV (comma separated value) file
• Import the CSV file to Elasticsearch
• Export images from TMS and upload them to S3 (Amazon Image Hosting)
• Tile images by IIIF specifications
• Explore the data in Elasticsearch
• Create a dashboard in which technical and non-technical museum staff can check on the status of each of this services
• Extract metadata from images using machine learning and computer vision techniques, and store this in Elasticsearch
• Have all of these things happen every night to take into account changes the museum staff may make to the data in TMS

THAT IS A LOT OF THINGS. How to we keep our code simple and manageable, following the single responsibility principle? How do we make sure if one piece fails, the whole system doesn’t come crashing down? How can we spread the computation out across multiple processes? In short: how do we avoid the backend monolith?

The answer is microservices.

Microservice architecture structures an application as a collection of loosely coupled services, which makes the architecture more modular, as well as easier to understand, develop, and test. It is especially nice for this project because it makes it easy to add new pieces later. Since each microservice has well-defined inputs and outputs, it means that if we add a new piece later— a new way to extract metadata from the collection images, for example—it’s easy to plug it into the existing system.

Enter Seneca.js

We decided to use Node.js for this project — JavaScript is currently the most popular programming language and we’re trying to build something that can be passed to a small museum technical staff (of a single person). Therefore, we chose a microservice framework for Node called Seneca.js. We had never used it for a project before, but it is actively maintained and straightforward to use. Here’s an adapted example from their documentation, a microservice to manage a todo list.

This is the core of a microservice. Calling seneca.add() creates an action, which is a string name bound to a function. By convention that name has two parts, a role and a cmd. To call the function, you use seneca.act() with the role and the cmd. Here’s how you would call createTodo() using an action:

Seneca makes it easy to share this function with other processes. With one line of code and a call to seneca.listen(), all of the actions we have created with role:todo become available. If you run node todo-pin.js, this starts the todo microservice listening for TCP connections on port 10101.

With that microservice running, all we have to do is change one line in server.js. Instead of calling seneca.use() to include the microservice core, we call seneca.client() to bind to the microservice. Figuring out which TCP port to connect to, creating and managing the connection—Seneca does all that for us. All we have to think about is actions.

One dashboard to rule them all

Now that we’ve seen the basic structure of a microservice, let’s look at how we’re using microservices in this project. One of the things we wrote is a script to export data in TMS to a CSV file. We wanted to make it possible for any non-technical user to run that task and check on its status. We also wanted that same user to be able to see the state of all tasks at a glance. So, we decided to write a dashboard website to manage backend tasks from the browser.

Wrapping the TMS to CSV export script in a microservice made it extremely easy to control that script from the browser. Here’s how that might look, mapping Express.js routes to Seneca actions:

In the dashboard, a button can be mapped to the AJAX request GET /api/tmstocsv/run, which then calls the Seneca action role:tmstocsv,cmd:run.

As a sidenote, this is such a common pattern that Seneca includes an Express adapter that binds Express routes to microservice actions automatically. (We use this, it’s great.)

Pulling it all together, we’re able to present a dashboard website with the status of all tasks in one place.

A not so beautiful but functional dashboard

Organizing our code into microservices has made it easy to maintain, read, and test. Not only that, the microservice architecture helped us write a task management dashboard that anyone at the Barnes Foundation can use. In our next post we’ll discuss how we give museum staff better insight into their collection data, using Elasticsearch and Kibana.

The Barnes Foundation collection online project is funded by the Knight Foundation and our code is open source. Follow the Barnes Foundation on Medium.