🌒 Shadow requesting for great good

Build confidence in how your web applications perform under additional load, safely

Christian Gregg
Dec 13, 2019 · 4 min read
Graph showing shadow (purple) vs real (orange) requests during a recent test

So what is shadow requesting?

At its core, shadow requesting means replaying live requests shortly after they have completed. Either to the same host (in order to simulate additional load), or to a different host (to verify parity of responses).

Why shadow requests?

One of the benefits of load testing via shadow requests is that we are simulating real traffic patterns without any configuration overhead; by replaying the requests almost in real-time we create a lifelike scenario of additional users on our systems without needing to maintain a repository of user flows and their associated HTTP requests.

How are we doing this?

We developed a tool, umbra, to facilitate shadow-requesting. It works as a Rack middleware, which sends incoming requests, via a Thread Queue, to a Redis pub/sub channel. In a separate process (or processes), a subscriber listens to that same channel and replays the request.

What do we need to watch out for?

We currently configure umbra only to shadow idempotent requests. As a general rule this means only GET requests, however, we also use an allow/deny list of endpoints in order to ensure we are not inadvertently manipulating production data.

What are some of the outcomes?

We recently tested a new middleware using umbra. One of our applications receives around 50% of its traffic from internal clients. We already set client-side timeouts on internal requests in order to mitigate against cascading performance degradation.

Results of even/odd test of load shedding on shadow-traffic only, purple lines depict those requests processed with load-shedding enabled, orange without.
  • Our Apdex score was 10% with load shedding enabled
  • P95 and P90 request queuing was significantly lower with load shedding enabled (and less significantly for P99)
  • Server error rates were around 1/2 for requests being processed with load shedding enabled

carwow Product, Design & Engineering

What happens under the hood at carwow.co.uk

Christian Gregg

Written by

Software Engineer @ carwow

carwow Product, Design & Engineering

What happens under the hood at carwow.co.uk

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade