Building a versioned UI deploy system for fast, stable deploys and rollbacks

At Blend, we’re working to bring simplicity and transparency to consumer lending. In the last two years, the Blend engineering team has doubled from 50 engineers to more than 100. Unsurprisingly, our codebase has grown in size and complexity as well.

As the team has grown, we’ve embraced the principle of distributed ownership. On the UI team, this means owning our own technical health, and, as of a few months ago, our own release.

The problem

Because of the sensitivity of the data we work with and the consumer finance industry’s stringent compliance requirements, we have an extremely high bar for quality. Our UI consists of three separate apps that are housed in a monorepo along with our core backend code. Too often, when our UI code was deployed with the rest of the monolith, a regression in some unrelated part of the platform would block release for the whole platform. We wanted to be able to validate and release UI changes and bug fixes quickly, be blocked only by our own regressions, and revert changes quickly if we realized that we had shipped a bug without needing to rollback the entire app.

This meant we needed to decouple our UI release from the core backend release. However, decoupling the UI from the core backend introduced complexity that did not exist when the UI was part of the monolith. With a decoupled UI, we needed a way to ensure that the backend is serving the correct version of the UI; we needed to maintain stability despite being on our own release schedule; and we needed to ensure that frontend and backend changes are backward compatible.

Using a push model for UI version updates

Because Blend is a white-label consumer finance platform, we have a multitenant architecture, with a separate tenant for each customer (where a customer is a lender like Wells Fargo or U.S. Bank). We have several environments — dev, sandbox, preprod, beta, and prod — with varying degrees of required quality from a UI perspective. Within each environment, we have bundled tenants into customer groups based on each customer’s stated risk tolerance and desire for the for the latest functionality (e.g., Early Access (EA) and General Availability (GA)). All tenants within a customer group serve the same version of each UI app.

When our UI code was deployed with the rest of the monolith, the frontend and backend code were always on the same version. Now, as part of the decoupled UI release, we upload the frontend assets to S3. Then, when the server needs to serve up a UI, it fetches the app index file and serves up the HTML. We needed a way for a tenant to determine which version of each UI app to serve.

Our initial solution for a decoupled UI release for a given UI app (for example, the login app) was a pull model. First, we upload the assets (js, images, index file, etc) to S3. Then, when we want a customer group (for example, EA Beta) to start serving the new version, we send the version (login/v1.25.0) and location of the index file on S3 to a microservice that stores that version data for the customer group. Then, when an EA Beta tenant needs to serve up the login app, it would fetch the version data for its group from the microservice, then fetch and serve the index file from S3.

The problem with this model was that the microservice would become a single point of failure. Instead, we implemented a push model — so when we deploy a UI app for a given customer group, we send the version data to each tenant in the group, and the tenants are responsible for storing it in their own databases. Then, when the tenant needs to serve the UI app, it gets the S3 location of the assets from its database, then fetches and serves the index file.

In addition, we have a microservice that simplifies version management by displaying app versions for each customer group and exposing any tenant versions that differ from the rest of the group.

Maintaining stability through validation

All code merged to the monolith has to pass unit and end-to-end tests, but we added an additional requirement for merging frontend code to ensure that anything merged to master is prod-ready: pre-merge QA.

Most user-facing changes have to be tested by a member of our QA team before merge to fill any gaps in our automated testing and engineers’ spot-checking. In addition, because the UI release process is now totally decoupled from the core backend release, we needed a way to ensure that the new version of a given UI app would be compatible with the current version of the backend, and vice versa on backend release. For this, we wrote Pact tests for critical interactions, which we can run on cut between the release candidate UI app version and current prod backend version (and vice versa on backend deploy).

Once the Pact tests pass, we consider the two versions to be compatible and store this information. Then, during the beta and prod UI deploy jobs, we check that the release candidate has been validated with the current prod backend version, and abort the deploy if not. This allows us to keep our product stable while drastically increasing our UI deploy speed.

Ensuring backward compatibility

Decoupling the UI from the backend introduced the need for our code to be backward compatible, something we didn’t need to worry about when it was deployed with the rest of the monolith.

Since our frontend code is still in the same repo as the core backend code, it was initially hard to remember that a breaking API change could cause major issues if the corresponding frontend change was not deployed at the same time. To keep engineers cognizant of this, we now ban merging frontend and backend changes in the same PR, which forces us to make our changes backward compatible. This has helped us to maintain stability while gaining all the advantages of a separate UI release.

Benefits of a decoupled UI release

Thanks to pre-merge QA, E2Es on merge, and our trusty and speedy Pact tests, we’ve removed the need for any manual validation during release, allowing us to push fully-validated UI code from master to prod in about an hour (down from about two days). This has been a game-changer for dev productivity.

Additionally, serving the app index file from a URI stored in a tenant’s database makes for much faster rollbacks. To roll back to an older version of a UI app, we use our version-tracking microservice to send the older version data to a customer group. Each tenant stores the updated data and immediately begins serving the older version, all with no app redeploy required.

Another benefit to serving the UI from S3 is that during backend development, backend engineers who rarely touch the UI can run the app locally without starting up the frontend, which minimizes dev environment friction. On merges to master, we upload the new master assets and send the updated S3 location to our version-tracking microservice. In dev mode, we do use a pull model for version tracking — when running the app locally, the backend serves the master version of the UIs directly from S3.

Looking ahead

Owning our own release as a UI team has allowed us to move faster, not be blocked by unrelated regressions, and have the engineer with the greatest context on the new functionality run release, monitor for bugs, and roll back quickly if needed.

The foundation we’ve set up for ourselves by owning our release will allow us to iterate faster and continue to deliver high-quality, stable features to our customers quickly.

This latest evolution of our UI release is just one of many projects we’re tackling at Blend. If you’re interested in learning more about the work we do and want to join a growing team, check out our open opportunities at our San Francisco and New York offices.