Automatically detect and understand extreme events in real-world data When analyzing data, we often think about high-level metrics — how much revenue did we generate yesterday? How many users queried this API end-point? How many times was this product purchased? But sometimes these high-level metrics can become heavily skewed by…

Ten steps to ensure your data monitoring solution is effective. Every time a data alert fires (or fails to fire), one of four possible outcomes occurs.

Part 1 — Monitoring the quality of structured data at scale To compete in a data-driven world, organizations must consolidate data into centralized warehouses and use it to enhance products and inform decisions. Data is now a strategic asset. But how can organizations ensure they can trust the data underpinning…

How to build and maintain high quality data without raising billions Airbnb has always been a data driven company. Back in 2015, they were laying the foundation to ensure that data science was democratized at Airbnb. …

When testing data, our first instinct is to reach for perfection. Can’t we write down a clear set of rules that govern exactly how our data should behave, just like we do when testing software? Of course we can’t! Data isn’t software, and shouldn’t be tested in the same way.

When we think of data quality, the first issues that come to mind are visible problems like duplicate rows, NULL values or corrupted records. But in fact the most common data quality issue is that data has simply disappeared. In this post we will describe how data disappears, what the…

Know your model’s limits, interpret it’s behavior and learn from variable length sets. At NIPS 2017 what surprised me the most was not the size of the crowds (they were huge), the extravagance of the parties (I sleep early) or the controversy of the “rigor police” debate (it was entertaining). …

Analyzing 700 women founders, and the VCs that invested in them. On Friday, I awoke to a trickle of twitter updates about an article in The Information, detailing the unwanted advances of a VC, Justin Caldbeck of Binary Capital. The article was shocking, with three women on the record (and…

Grocery delivery visualized in python with datashader. At Instacart, we deliver a lot of groceries. By the end of next year, 80% of American households will be able to use Instacart. Our challenge: complete every delivery on-time, with the right groceries. Over the course of a week, we traverse cities…