End white-knuckle releases with Backplane’s blue-green deployments

Mux, a video performance analytics company, monitors the video streaming performance of hundreds of millions of streams. Ask any engineering team and they will agree: code deployment and feature releases can be incredibly nerve-wracking.

- Adam Brown, Head of Infrastructure Engineering at Mux

Leading digital companies including Funny or Die and PBS depend on Mux for critical infrastructure. As a modern, global business uptime is crucial to their success. Funny or Die’s audience is 70 million+ people worldwide. If Mux goes down, Funny or Die’s video quality could degrade significantly. Mux’s data shows that “just one buffering event decreases the amount of video watched by 39 percent.”

That’s why the typical strategy of hitting release, watching the error logs, and holding your breath doesn’t work for companies like Mux. They couldn’t afford to white-knuckle every deployment. They needed to deploy with confidence.

Mux turned to Backplane’s blue-green deployments. Blue-green deployments allowed them to move traffic progressively, not at all once, as they developed confidence in new code.

If issues started to pop up, they would be able to roll back the traffic, minimize the damage and prevent a small issue from becoming a big one.

What are Blue-Green Deployments?

Blue-green deployments consist of two pre-production environments labeled blue and green. Only one environment is live at any given time — let’s say that you’re live in the blue environment. In the green environment, you can test out changes, make updates, or tweak your setup to get ready for prod.

Once changes are ready to ship, simply switch traffic from blue to green. You can shift traffic to green incrementally to ensure things run smoothly, rather than shifting all of your traffic at once. If errors start to occur, you can minimize the damage by rolling-back your traffic to blue until the issue is resolved.

By gradually shifting requests, blue-green deployments minimize the risk of customers experiencing downtime. In the animation below, the stream of requests is switched between the blue and green environments starting with five percent and moving incrementally up to 100 percent of traffic.

Here’s how it works:

1. Requests are sent in from the user and matched with an endpoint. Endpoints can look like example.com, api.example.com, or www.example.com/blogpost. If a request for http://www.example.com/blogpost/archive/hello-world is received, Backplane will match it with the www.example.com/blogpost endpoint.
2. From each endpoint, you will end up on a blue route or a green route. Each route has a weight that determines what percentage of traffic it receives.
3. Requests are then directed by the route to a backend, either blue or green, corresponding to the route. Backends are HTTP web servers connected to Backplane that respond to requests.

By adjusting each route’s weight, you shift traffic between your blue and green routes and backends. If errors start cropping up upon deployment, dial the blue route’s weight back up to 100 and the green’s back down to 0. Otherwise, progressively dial up green’s weight until you’ve shifted all of the traffic.

How to Set Up Blue-Green Deployments in Backplane

Blue-green deployments are a subset of the general problem of shaping traffic across different routes and endpoints. It’s about shaping traffic in the context of deploying new code. If you’re already using Backplane to route traffic, Backplane offers blue-green deployments as a feature.

“Often it’s very difficult to do that routing of traffic, but Backplane makes blue-green deployments trivially easy,” said Adam Brown, Head of Infrastructure Engineering at Mux.

Here’s how.

1. Label your backend with its version number

By labeling our backends by version number, we’re able to group those backends together and assign them to separate routes, one set blue, the other green.

When a backend connects to Backplane, it provides a set of key=value pairs called labels. For backends you want to group together as your blue backends, run the following command:

$ backplane connect "endpoint=example.com,release=v1"

For backends that you want to group together as green, run the following command:

$ backplane connect "endpoint=example.com,release=v2"

Now you have two separate, non-overlapping sets of backends to route traffic.

2. Create separate routes based on version number

Next, create two separate routes, one to release v1 and the other to release v2.

Run the following command to create a route from endpoint example.com to all backends labeled with name=example,release=v1. This is your blue route that feeds your blue backends.

$ backplane route example.com "name=example,release=v1"

This command will create a route and call it something like route0001.

Run the same command for release=v2 to create the green route.

$ backplane route example.com "name=example,release=v2"
{PLACEHOLDER CODE}

The outcome of these two commands means that you have two routes, route0001 and route0002. By shifting traffic from route0001 to route0002, we'll be moving it from release=v1 to release=v2, until v2 is 100% released for all traffic.

3. Progressively shift traffic from one route to the other

Shifting 100 percent of traffic right off the bat from route0001 to route0002 is a single command. This is in effect how a lot of companies release in the absence of blue-green deployments.

$ backplane shape example.com route0001=0 route0002=100

A second ago, 100 percent of traffic went down route0001. After the command, 100 percent goes to a new version.

Let's say instead you want to switch traffic from route0001 to route0002 in increments of 10 percent . You would start like this:

$ backplane shape example.com route0001=90 route0002=10
$ backplane shape example.com route0001=80 route0002=20
$ backplane shape example.com route0001=70 route0002=30
$ backplane shape example.com route0001=60 route0002=40
$ backplane shape example.com route0001=50 route0002=50

At this point, we’ve shifted 50 percent of traffic over to route0002.

In the case that something goes wrong, you can easily rollback all of the traffic to route0001 (while you work out the kinks) and then redeploy.

$ backplane shape example.com route0001=100 route0002=0

If everything is operational, you can finish up with the following:

$ backplane shape example.com route0001=40 route0002=60
$ backplane shape example.com route0001=30 route0002=70
$ backplane shape example.com route0001=20 route0002=80
$ backplane shape example.com route0001=10 route0002=90
$ backplane shape example.com route0001=0 route0002=100

With the last command, you have 100% of traffic flowing to your v2 release with minimal downtime and disruption to your customers.

Confidence is Contagious

Backplane’s blue-green deployments allowed Mux to progressively move traffic to a new service rather than moving it all at once. With Backplane’s blue-green deploy feature, Mux can deploy more frequently with confidence. Backplane has eliminated the high stress event of releasing an update for the team.

In the midst of rapid scaling — attracting nearly $12 million in venture funding in just over 18 months — Mux needs to focus on serving a rapidly growing customer base and innovating on the problem of video metrics collection. It can’t afford to be outpaced by the competition, or get slowed down by a lack of confidence around new version releases.

Today, Mux bakes blue-green deployments into nearly every deployment. They save time by using Backplane to run their blue-green deployments since they don’t have to build and maintain an internal tool. Fundamentally, blue-green deployments have helped the entire engineering organization innovate and move faster, with ease.

End your white-knuckle deploys with Backplane today.

Looking for startup speed at Enterprise-scale? Reach out and we’ll set up a free 14-day trial. And Backplane will be securely working for your teams, in less time than it took to read this blog.