Building Systems With Static APIs
APIs are inherently dynamic, but it doesn’t mean every part of the API infrastructure must be dynamic. The “Static API” concept is an alternative data engineering pattern, which can benefit you, and possibly your wallet.
Static APIs can:
- Reduce cloud computing costs.
- Simplify data architecture & management.
- Scale faster than dynamic solutions.
What is a Static API?
A static API consists of pre-generated assets. Many APIs transmit data as JSON or XML, which can be pre-generated. They can be stored as simple files on a server, object store, or CDN as static assets. In crude terms, a static API serves pre-generated responses.
Retrieving Static Data
By retrieving static assets, runtime database read operations are eliminated. An API request doesn’t need to query a database to serve a response. Instead, results are served from the collection of static assets - often times from a cache.
If this sounds familiar, you may be a web engineer recognizing this is basically how static websites work. Static websites are one of the easiest and most mature forms of serving content, as well as one of the most efficient. At the most basic level, you just need a web server and some files to serve.
Precedence: Static Website Generators
In web development, static site generators are more than just a trend. They’ve proven to be an efficient use of computing resources. It’s based on the core principle of generating HTML pages that can be served as static resources, rather than dynamically rendering a template on each request.
While static site generation may feel like a step in the wrong direction, there are many prospective benefits to this approach. Aside from speed, performance, and version management, this approach can also reduce costs. For example, a virtual machine usually costs more than a static host. Static hosting is a commodity service with many inexpensive choices.
Static sites cannot generate custom views dynamically without dynamic data. Developers circumvent the data challenge using the JAM Stack approach. This approach leverages a static site and a dynamic data API to create great user experiences. This is a great way to separate concerns (rendering vs data). Nevertheless, it merely shifts the infrastructure burdens from one system component to another.
For many sites, there may be a better way to leverage your computing resources.
Consider Using Static APIs
A static API is conceptually similar to a static HTML page. The core principle is simple: generate data files like JSON, XML, and other formatted files. Instead of dynamically generating JSON in response to an API request, responses are pre-generated and served like a standard web page. The concept is similar to caching. It stores pre-generated data. It also adds version control, immediate flushing, more visible control, etc.
The major added benefit this approach provides is an alternative scalability and workflow model.
It’s often easier and cheaper to push static content to network edge locations in the same manner a CDN would. As a result, applications can leverage commodity resources - including existing CDNs. This can be substantially cheaper than running and scaling APIs on virtual machines (especially during traffic spikes).
The workflow also changes. For example, I have become fond of using a centralized relational database to fire change events. These events trigger a process which updates the static JSON files. Processing is only done “as needed”. When the database isn’t inundated with queries, the processing load is significantly lower.
There are, of course, some obvious considerations that must be addressed with this approach. Security is the most important. Instead of doing per-request authentication/authorization at the database level, an API gateway can be used to restrict access to sensitive static resources. This expands enforcement capabilities with security features like CORS and rate limiting.
Example
At Metadoc, documentation is treated as a data source. It is a great use case for a static API. Metadoc uses EnableDB to interact with its data persistence layer. EnableDB captures database change events as they happen, allowing static asset updates to be performed whenever a change occurs.
In simple terms, an update will write data to the database. This triggers an update to the static JSON file. Further API requests are served statically.
This is similar to database triggers or Kafka event streams, which trigger whenever a change event occurs. EnableDB sends a notification to our processing layer with the data changes attached. Our processing layer pre-generates or updates existing JSON without querying the database. The result is served to end users through our CDN.
Benefit: Modern CDN’s update static assets at edge locations around the world in near real-time. It’s like your data exists wherever your users are without having to run database servers all around the world.
The net effect of this architecture is an API that serves content statically while still being dynamic. Most applications overwhelmingly read from databases, while write operations make up the small remainder. EnableDB facilitates a “write only” approach. This means there is far less interaction with the database.
From a financial perspective, this approach reduces the queries and processing occurring on database servers. Since fewer demands are placed on these servers, fewer computing resources are needed. This results in lower operating costs. Serving pre-generated content uses some of the cheapest cloud commodity services possible: basic file storage.
Simplified Workflows & DevOps
While the diagram above may not appear simple at first glance, it is pretty easy to maintain. The static API approach separates concerns.
For example, if an organization is using a database with an expensive license, it is possible to swap out the database without having to rewrite connections and logic for an entire API. If the database goes down or needs maintenance, the static API will still be available (even if it’s a little outdated). If someone mistakenly drops a database table, the data could potentially be restored from static assets. When testing new API endpoints, mock JSON can be served while database updates are being constructed - without downtime.
Static APIs shift processing, making it possible for different teams to work on the parts of the data pipeline they are best at. Web teams can focus on the UI. API teams focus on the endpoint design. Database Administrators manage the database. All of these are separate functions within an organization. The static API is well suited for letting each expert focus on what they do best without impacting others.
Scaling
Scaling a database cluster is hard. Many people pay cloud providers a significant amount of money to do this for them. However; many have come to realize that scaling databases on cloud infrastructure can get extremely expensive.
On the other hand, the world has made a science of scaling static content. Trillions of static web page views are processed on the internet every day.
Static APIs use the best of both worlds. Database management is simplified while web hosting strategies are used to scale the most demanding parts of applications.
When Static APIs Aren’t Right
There are scenarios where a static API may not be the best fit.
If an application requires extremely low latency responses, a static API may not be an appropriate fit. Then again, most databases aren’t an appropriate fit for this either. To determine if a static API is appropriate for you, ask yourself how much latency is acceptable between the time when the update is made and the time the API serves the updated response.
For example, at Metadoc, when a user updates data, the user gets an immediate response. However; it may take 1–2 seconds for the new JSON to be available and the CDN cache to be updated. This means the updated content will not be available to other users for 1–2 seconds. For a documentation platform like Metadoc, this is completely acceptable. For a stock trading app, latency like this may be the difference between a great deal and a bad one.
Experimenting With Static APIs
It’s not difficult to experiment with static APIs when writing them by hand. The only requirement is a web server.
Any web server will do, but I favor Fenix Web Server when prototyping new static APIs. I start by writing an initial JSON file by hand, using Fenix to serve it for my JAMStack apps. I’ll test, play around, think about the app, etc. Once I’ve settled on a data format, I’ll write a script to generate JSON files in the web server directory. This is a great way to simulate how a production static API would function.
Building a full data processing pipeline will ultimately require more effort, but getting started is very simple.
If you try this out, leave a comment about your experiences. If you’d like help building a production static API/data pipeline, reach out to me at Butler Logic.
If you liked this article or the concepts presented, clap for it (it’s how I determine what else to write about).
