Why Quorum’s engineers will never have to write an if-else statement again

The components of Quorum’s stack are each the best in their respective categories. PostgreSQL is the most powerful and mature SQL database, Django’s ORM is elegant and friendly, and React is the poster child for cutting-edge Javascript. Choosing these unrelated technologies has proven to be the correct decision time and time again, but making them play well together has occasionally been tricky. Using different languages on the client and server has prevented Quorum from becoming an “isomorphic” application, in which the same code executes on both. However, isomorphism has been the inspiration for many of Quorum’s most creative efforts to connect the pieces of our stack, including cross-language surfaces, an abstraction of REST, and a publication system for Django.

typedef Enum

One problem that shows up frequently in large programming projects is how to most effectively enumerate a discrete set of options. This is particularly important for Quorum, whose database contains hundreds of thousands of people, millions of bills, and nearly a hundred million documents. For each of these there are many things we may want to enumerate, such as the type of media a document contains, the status of a bill, or the gender of a Member of Congress (while Quorum supports all identities, Postgres does not yet understand gender fluidity).

Consider trying to model the political party a member of congress represents. Without the ability to enumerate, one might solve the problem like this:

Under the hood, Django would create three columns in the Member table, and for each row one would be true while the other two are false. This would work, but is not scalable. For each enumerable property you would need n boolean fields for n possible options. Suppose that instead of people we are modeling documents. If Quorum scrapes 100 types of documents we have to add 100 boolean fields to keep track, which is 100 additional columns in our table of 100 million rows. That amounts to 10 billion cells, which, even for single-bit booleans, adds 1.25GB to our AWS spend for a single property, of which we track hundreds. We would have gone bankrupt within a month.

In general, the pattern computer scientists use to solve this problem is an enum. While many languages such as C, Swift, and OCaml have enum primitives built directly into the language, Python does not, and its standard implementation leaves a lot to be desired. However, django-common contains an implementation specifically designed to work with Django’s ORM and the underlying SQL.

This is much better. Instead of multiple booleans we can have a single integer field, whose value corresponds to one of the choices in the enum. Quorum quickly developed hundreds of enums and added integer fields to many tables throughout the database.

Alakazam?

This solved the problem of storing enumerated properties in a SQL database, but things got a little ugly when trying to build a web application on top of it. Quorum’s frontend is a Single Page Application that uses React and Redux to render data served through a REST API (read more here).

Consider the attempt to render the image in this component, taken from Quorum’s search page:

In order to create an image with a red border (or blue for democrats), we need to render animg tag with a class corresponding to the party the member belongs to, so that we can apply css rules to give the circle a border of the appropriate color.

Javascript has no enum primitive, and more importantly does not have access to the declarations in Python that determine what the integers mean. As a result, the burden is shifted onto the engineer, who must constantly cross reference with the Python files to hard code the correct integers in Javascript. Quorum’s frontend quickly became riddled with magic numbers and nightmarishif-elsestatements. We faced bugs from one out of the 50 cases being mismatched, duplicated, or forgotten. Even worse, to change or extend one or more of our enums (from adding a new type of document, or expanding to new regions), we had to find every use of the integers in that enum throughout the site and change the values or add the additional cases. If you’ve ever tried searching a codebase riddled with magic numbers for “1” or “14” you might also be inspired to do something creative.

Isomorphic, sorta

What we needed was a way to access the information stored in our various enums in Javascript as well as Python, without having to rewrite them.

By utilizing the inner working of django-common in conjunction with some Python fundamentals, we found a way to JSON-serialize all of our enums so that they could be sent to the frontend.

We used a context processor to “publish” this JSON to Django’s template renderer, and then injected it into Javascript memory in our base template.

With our enums stored as JSON in Javascript memory, we could now design the way we wanted them to behave in Javascript. Inspired by the concept of isomorphism, we aimed to make them behave exactly as they did in Python, so that server and client code could be the same.

The result was a class that exposed the two most important functionalities: translating an integer into something semantically useful via by_value, and iterating through all potential options. Our image component could now be rewritten with a single expression with no magic numbers.

In this way we achieved a sort of isomorphism between the two layers of our stack. Any changes to enums would percolate to the frontend without our needing to change or cross reference any Javascript. Database records would contain integers, and the information necessary to decipher them would be available with the same surface on both client and server.

enum.MetaEnum

This completely changed the way we code, and allowed us to purge all of the terrible if-else statements throughout our frontend. From here we began to build on top of this system, and to use enums to describe higher order information about our system, such as the types of data our database contained and what could be done with them.

This proved extremely useful, and quickly expanded to include our 70+ types of data, with each item containing over 20 different values describing the way it should behave on both the backend and frontend. For example, there is only a certain subset that we want to be visible along the top of our Search feature.

Instead of writing out each button, we could iterate through the types of data and only show the ones that we flagged in Python as being searchable. We could store the integer corresponding to the current type of data in Redux, update it whenever a button is clicked, and allow React to do the rest.

REST Easy

Magic numbers from enums weren’t the only thing we were hard coding unnecessarily. In order to query information from our REST API, we were having to write the endpoints of our models over and over again. Even if we could cleverly render the header of Search, to fetch the results we’d have to construct urls by hand.

This would not scale to our 70+ types of data. We decided to apply the same idea to our models — JSON serialize a representation of our schema and publish it to the frontend, where it could be wrapped in a Javascript surface that would emulate Django’s ORM.

Ultimately, this Django query:

Person.objects.filter(party=Party.democrat)

Could be achieved in Javascript as:

Person.objects.filter({ party: Party.democrat.value }).GET().then(...)

We couldn’t achieve 100% fidelity due to discrepancies between Javascript and Python, such as treatment of primitives and keyword arguments. Additionally, while server-side Django has the luxury of laziness, the Javascript requires that the engineer specify when to send the HTTP request to perform a given query.

For a feature like Search, where we had been hard coding the endpoints for each model into separate action creators, we could now express the entire feature as a single function.

A State of Stack Synergy

With this system in place, the different pieces of our stack have finally begun to play together, yielding some incredibly powerful results. For example, we added the names, types and metadata of the fields on our models to the JSON dump, and used them to build our most recently released feature, Sheets. Sheets “autodiscover” the data in our database and present it in a tabular, customizable format to the user.

Drawbacks

While it proved tricky to emulate this functionality within our Javascript unit tests and our iOS app, the only real drawback we’ve experienced is a slightly heavier base template, which must render the JSON representation of our schema on the client. In total it’s only a few kilobytes the user must download once during their session, and we’ve been able to delete kilobytes of code they would otherwise have to download instead.

Isomorphish

Quorum’s unique stack combining Django and React has forced its engineers to constantly be creative in the ways that we code. While the industry’s standard of isomorphism isn’t achievable with so many disjoint pieces, our efforts to emulate it have proved worthwhile. This pattern of maintaining surfaces between languages and publishing schematic information has allowed us to eliminate lots of if-else statements, trivialized powerful abstractions, and made Quorum forward-compatible.

Are you a creative coder? Consider applying to work at Quorum.

Disclaimer: The code in this article is for illustrative purposes only. It has been editorialized from the Quorum codebase, and is not intended as a plug-and-play implementation of the concepts described in this article.