Optimizing voltapp-flask

Recently, I made some changes to optimize page load times (and bandwidth consumption!) for voltapp-flask, the web application that powers our demand response platform. They’re more in the “low hanging fruit” as opposed to the “expansive multi-month cross-functional effort” category, but I thought they might still be interesting to talk about.

Like many SaaS providers, Voltus uses a hybrid monolith/microservice architecture. We use microservices where appropriate to shorten development cycles and increase resilience (for example, most of our actual market integrations are written in Go and run as separate jobs in our nomad cluster), but we have a core application that allows access to our primary postgres database (via a REST API) and serves our React-based frontend.

This core monolith is called voltapp-flask. It’s written in Python using the Flask web framework (hence the name). This is by far the oldest part of Voltus’s platform but has generally held up pretty well over time. As is common with monoliths, there are some questions of ownership, reliability and scale but we’re on the way to solving them. My experience is that most problems of this type can be dealt with as one offs — incrementally improving, optimizing and simplifying an architecture. Like water flowing over a stone, over time you eventually end up with something smooth and refined.

Enabling response compression

One of the most basic things you can do out of the box to increase performance is to enable response compression (either at the application layer or via a load balancer). If you’re not doing this, you can (in some cases) literally be making your users download *megabytes* of unnecessary content: JSON API responses often contain a fair bit of redundant information, and thus compress very well. External user responses are usually reasonable in size, but some of our internal users (for example, our dispatch operations team) need to work with very large sets of data — often over a metered and/or slow connection if they’re on the road.

I noticed that voltapp-flask wasn’t using response compression accidentally– I was adding this feature to some internal API services and wanted to show an existing example of response compression at our engineering weekly demo day. Looking via the Firefox network panel at a common endpoint (our real time energy dashboard), I was surprised to find that the transferred size was equal to the response size for our JSON responses and that the content-encoding header was not present!

Fortunately, fixing this was easy! There’s an excellent package that you can easily add to provide response compression called flask_compress, which I just slotted in. The diff is literally just a few lines:

diff --git a/voltuspy/voltapp-flask/voltus/app.py b/voltuspy/voltapp-flask/v
oltus/app.py
index 4aaf053a2e..b3d898fa12 100644
--- a/voltuspy/voltapp-flask/voltus/app.py
+++ b/voltuspy/voltapp-flask/voltus/app.py
@@ -7,6 +7,7 @@ patch(requests=True)
 import os
 import logging
 from flask import Flask, g, request, Response
+from flask_compress import Compress
 from config import config
 from time import time
 from voltus.utils.json import VoltusJSONEncoder
@@ -54,6 +55,8 @@ worker_logger = get_celery_logger_by_name("celery.worker")
 
 initialize_flask_server_debugger_if_needed()
 
+compress = Compress()
+
 
 def create_app(config_name=None):
  """Main application factory."""
@@ -67,6 +70,9 @@ def create_app(config_name=None):
  init_celery(app)
  init_cli(app)
 
+ # enable compression
+ compress.init_app(app)
+

It’s also possible to apply this type of optimization at the load balancer level, depending on the details of your setup.

With this applied, requests to VoltApp API endpoints are compressed with the Brotli algorithm (supported by all modern browsers), and transfer volumes go way down. Here’s an example of an endpoint loaded in Firefox’s network panel:

After making this change, I made a quick announcement on it in the #general Voltus channel (mostly as a precautionary head’s up to ask the company to report any problems). Was pretty happy to see this feedback from the leadership team:

At Voltus, part of our culture is celebrating wins across the company (no matter how large or small). I think this comes from the founders’ sales background, though I’m not totally sure. Regardless, it feels really nice.

Remove unnecessary steps

This next step was a little more involved and has less dramatic benefits, but is still interesting as an example of how to reason about the whole system.

Our frontend React-based pages (static HTML) are served up via a small microservice called a “template server” (using Express under the hood) which is called into by voltapp-flask whenever a page is requested. This architecture lets us decouple regenerating the JavaScript assets and updating the frontend from deploying voltapp-flask. To avoid a few round trips on the client, we inject some data (like the user permissions) into the frontend before sending the response back.

This approach has a lot of advantages (credit to Jamie Charry for thinking it up) but it had one weakness: to get the extra metadata, it needed to call back into our monolith. This was generally not that expensive, but it does introduce some extra latency (anywhere from 25 to 75 ms, depending on various factors). This is just on the edge of human perception, but since all other frontend operations block on retrieving this initial data, it seemed like a decent place to put in a little bit of extra effort. You can see the effect of this by looking at this datadog trace:

The purple “express.request” illustrates the bottleneck here when processing a request. All of this work is unnecessary: voltapp-flask already knows what permissions the user has. I did up a pretty simple patch to move the variable injection (a string replace) back into voltapp-flask. This caused the purple section above to vanish from our traces, and I was immediately able to see a nice bump in our performance traces for the template server service:

Conclusion

In the software industry we like to talk about “grand solutions” to performance problems, but some simple actions can often yield a significant benefit at minimal cost (both in terms of implementation and diagnosis). Don’t neglect to work on and celebrate the basics: they’re the foundation of a delightful product that’s a joy to use.