Engineering a successful product launch

How not to do a big bang that causes a big bang

James Stanier
Nov 22, 2018 · 7 min read
Photo by SpaceX on Unsplash.

Panic, panic!

The engineering team looks frantic. There are empty takeaway coffee cups across their desks, in the bin, and on the floor. Kelly is slouched over her keyboard looking at her monitor through her fingers. It’s the loading prompt, and it’s still loading, even after three minutes.

“This just can’t be possible,” she remarks. “How come we’ve never had any issues with the loading speed until now?”

A Slack message from QA.

Ash: Why is it throwing an Internal Server Error every time you change the date range to last month?

Kelly: What?

Ash: Try it out, see if it happens for you too.

Kelly: ARGH, let me have a look…

Bringing up the Chrome developer console, she feels the subtle change in air movement as someone approaches her.

“I’m really sorry to interrupt you, K.”

It’s Evan, the infrastructure engineer. Kelly tries not to get frustrated and scoots back on her chair.

“What’s up?”

“The database just isn’t coping with the number of requests from your service. I think it might need reworking. When is this meant to ship?”

Kelly feels her cheeks flush red. “Tomorrow.”

Another Slack message pops up. It’s Marketing.

Jordan: I’m just about to send out the countdown video on Twitter. App doesn’t seem to load at the moment. What’s going on?

Jordan: Are you there?

Kelly feels the world spin, and would rather be anywhere else right now.

The most wonderful time of the year

Building software is hard enough in the first place. Building it to a deadline is even harder. Building it to a precise deadline where all of the company — and soon your whole customer base — is looking at you, is terrifying.

Nothing ever goes entirely right in the software. The bits of the project that you thought was going to be difficult to turn out to be straightforward and the bit that was going to be simple takes four times as long because of some obscure networking issue.

To top it all off, the Kraken-like mega problem that nobody could have predicted beforehand eats all of your contingency time, and here you are again, once again — and you will forever be here, no matter the project — fixing bugs and performance issues right before the deadline, overtired, over-caffeinated, and overstressed.

The big bang theory

What do I mean by big bang launches? I’m talking about launches where the application or feature:

  • Is shipped to production only as the marketing launch goes out.
  • Is enabled for all users at once.
  • Hasn’t been profiled against real load in production.

The attraction of the big bang launch to the outside observer is clear: it is the ultimate demonstration of the whole company being in tight formation. There was nothing, and now there is something big. Magic.

The engineering, the marketing, the salespeople, everything — all of the planets align at just the right time — and the curtain opens in front of the ballet to rapturous applause. What a display of coordination and synergy!

But, this just-in-time delivery never goes right. Ever.

What kind of strategy can we adopt for making it look like we’re performing miracles but we’re actually being measured and safe instead?

How can we allow all of the space that we need to roll things out in production, test them and tweak them, whilst still factoring in contingency in such a way that allows for flexibility when things inevitably go wrong? How can we do this with our existing and prospective users being none the wiser?

Engineering strategies for a smooth launch

  • Planning for usage that is many times beyond what your current system can handle.
  • Making use of feature flags in order to constantly ship the feature into production where you can see how it behaves.
  • Doing extensive load testing early enough to be sure of your architectural decisions.
  • Taking advantage of beta programs with trusted customers to get real, non-internal feedback on the production code as early as possible.
  • Using shadow loading to see the real production footprint, without those users knowing they’re using it.

Let’s visit each of these in turn.

Planning for usage

If it doesn’t, is it possible to scale it horizontally? How would that work? More applications, more servers? How will requests be routed? Round robin, or sharded by client or user? Is it possible to get away with estimates of data rather than needing to count and aggregate everything? Will values be computed in batch, real-time or pre-computed?

Get creative. There may be a really neat solution.

Asking these questions with a diagram on a piece of paper or on a whiteboard is easier, cheaper and less stressful than doing it a week before launch. I once read that most of the effort in system design should be on picking through the edge cases and the contingency plan, rather than trying to make the core design perfect.

Make sure that you have a clear route for future scale, otherwise, you’re going to be replacing the wheels on a moving car, rather than on a stationary chassis.

Feature flags

Feature flags, especially those that are highly customizable such as the ones provided by Launch Darkly, mean that you can test functionality with a percentage of your customers, or you can enable features to internal staff who will give you valuable feedback without the feature needing to be polished. You can also use flags to coordinate beta programs. When it comes to shipping time, you can roll out new functionality to customers in incremental cohorts to measure the impact.

Feature flags are great.

Load testing

What’s your 99th percentile case versus your 50th percentile? Is it acceptable? What will your biggest customer experience versus your mid-tier one?

Beta programs

With the help of feature flags, you can give them the unpolished functionality ahead of time, monitor how the system performs under real usage patterns, and also speak to them for qualitative feedback.

Implementing their suggested improvements makes the final product better for everyone.

Shadow loading

For example, if your new feature is going to be shown on the top of your homepage, why not have all users unknowingly call that new endpoint on page load, with the results being discarded?

This way you can measure what the load on the system is going to be like during normal conditions. You can feel assured that the functionality is ready for showtime.

In summary

By following some or all of the techniques above, you can ensure that on the day that you flick the big switch on for all of your users, the system has already been doing all of the work, predictably, for some time. You can go out for a celebratory lunch with the team without a feeling of paranoia that everything is about to blow up catastrophically.

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +391,714 people.

Subscribe to receive our top stories here.

The Startup

Get smarter at building your thing. Join The Startup’s +794K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

James Stanier

Written by

VP Engineering @brandwatch. Writing things that interest me. Hopefully they'll interest you as well.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +794K followers.

James Stanier

Written by

VP Engineering @brandwatch. Writing things that interest me. Hopefully they'll interest you as well.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +794K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store