Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash

Overview

Dispatch at DoorDash

  1. the core assignment problem — deciding which dasher is the best suited to fulfill a delivery, in an efficient, robust and scalable way.
  2. machine learning algorithms to predict the numerous timepoints in the life of a delivery — “when can this order reach the consumer”, “when will this order order be ready for pickup?”, etc.
  3. algorithms to decide how and when to group multiple deliveries headed in similar directions at similar times
  4. how to leverage various marketplace shaping strategies to balance supply and demand, including pricing changes
Illustration of the core assignment problem

Marketplace Experimentation

Switchback Experimentation

Implementing Switchback Experiments

Illustration of time-market unit randomization
  1. We randomize the variant used for each time window, rather than simply randomizing the initial variant and flipping back and forth deterministically. With this approach, each time unit is a randomized experimental unit.
  2. We further split across different geographical units (regions) and randomize them independently, so region A could use the current algorithm for one window and the new algorithm for the next, while region B could use the new algorithm for the first window and the old algorithm for the next. This in a sense actually creates a series of ‘time-region units.’
  1. Storing metadata on all experiments (Metadata)
  2. Choosing the algorithm variant to use at certain point of time for any experiment (Bucketing)
  3. Handling tracking to enable metrics calculation and analysis (Tracking)
  • Based on the time passed by the registration endpoint, the switchback service checks and updates its internal state e.g. check if it is time to update the buckets and if so, flip a (multi-sided) coin for every region and determine the bucket for the new time window
  • Used by the experimenter system to ask for the current bucket (variant) for an experiment
  • the switchback service fetches the bucket value from its state and sends a tracking event (which includes experiment metadata and unit identifier) to the ingest pipeline.

Analyzing Switchback Experiments

Setting up Experiments and Doing Power Calculations

Validation of our Switchback System

Conclusion

Appendix

Numeric Data for Sample of 14 deliveries used in example in article above

--

--

--

Get more from your neighborhood. Order now at www.doordash.com.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

When will deaths from Covid-19 stop in Europe

Data tour to Boston and Seattle..!!

Data Sandcastles and the Meaning of Existence

How data driven companies build customer personas

The clash between Data Scientists and Machine Learning Engineer, the winner is…

101 DATA SCIENCE Cheat Sheets (ML, DL, Scraping, Python, R, SQL, Maths & Statistics)

Just Rankin’ Sh!t

Digital Twins of Retail Stores: How to Optimize Your Staff Schedules

A snippet of monthly employee schedule with shifts and days off and in physical stores created using digital twins and pygmalios data

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
DoorDash

DoorDash

Get more from your neighborhood. Order now at www.doordash.com.

More from Medium

Feature scaling

A Machine Learning Framework for Predicting Purchase of Online Customers based on Dynamic Pricing

Hybrid Recommender System