An exploration of serverless architecture by means of a small web app that provides visibility into BitGo’s indexer infrastructure.
Benefits of Serverless Design
Motivating this project was a lack of visibility into our blockchain indexing infrastructure; tangentially, I wanted to use this as an opportunity to explore the world of serverless apps.
Coming from 10+ years of traditional server-side web development, this shift took time to wrap my head around. However, once I decomposed the problem from a new perspective, the benefits of serverless development became immediately clear:
- Reduced Infrastructure Overhead: Zero need to spin up web servers (locally or otherwise), configure database boxes, or download + manage traditional web app frameworks.
- Low Maintenance Costs: Once things are up and running — they stay running. There is little to no infrastructure cost and nearly zero maintenance involved once deployed.
- Architecture Simplicity: Building a serverless app forces you to constantly consider how things can be simplified — be it the product, the architecture, or the code. Instead of thinking of ways to add or extend functionality, I found myself thinking about ways I could prune it.
At it’s core, the app does four simple things:
- Inspects the status of BitGo’s indexer infrastructure via a set of public APIs.
- Compares the state of each indexer to that of the corresponding network via public block explorers.
- Serializes the state of each indexer to JSON and pushes that data to s3.
- Once on s3, this structured data is consumed by a lightweight single-page app that uses it render a dashboard, providing visibility into BitGo’s infrastructure health.
Traditional Approach vs Serverless Design
Building something like this in Node/Django/Rails typically requires decisions around a data stores + drivers, templating languages, hosting, load balancing, deployment, etc.
A serverless design approach, however, makes decisions much more simple:
- How will this app get the data it needs to render itself?
- How can a back-end service update the data that the app consumes?
Focusing on these two questions simplified all other decisions, and all non-essential concerns began to fade away.
Amazon has a fantastic service called AWS Lambda (there are analogous services on Google Cloud Platform and Microsoft Azure). Lambda allows you to define and execute functions without the need for permanent infrastructure.
Lambdas are invoked by “event triggers” (eg: a file is uploaded to s3, an event fires, and a Lambda is invoked to process the image) or “time based triggers” (eg: every 2 hours, invoke a Lambda to execute some custom logic).
Using a managed service like AWS Lambda, it’s dead simple to define functionality that checks the status of various services, compares those states, serializes the output to JSON, and writes that serialized output to s3.
Once you have a data store (in this case, s3) with a continuously updated feed of highly available data, it becomes trivial to build a static page that consumes it.
Even more powerful is the ability for this to be extended. Currently, we have a Lambda that writes serialized values to s3 and a web app that periodically polls for new data to render to it’s users.
If we wanted to do something like bulk process these JSON files to see how their values have changed over time, we could use AWS Athena or Elastic MapReduce to hammer through these JSON files to explore our data.
If we wanted to configure other triggers to send notifications to the team when an indexer falls behind chain head by more than 20 blocks over X consecutive checks, we could build another Lambda to process the most recently created JSON files, and wire that up to AWS SMS to send notifications.
The possibilities are endless and just as simple to implement using other AWS tooling. These design patterns feel akin to building small lego blocks and connecting them together to build really interesting, de-coupled functionality.