At Postmates, we believe that our cities and towns are our warehouses. We understand what’s available in a given market, index it all, and put everything your city has to offer at your fingertips. That means that at any given moment we’re monitoring the exchange of goods and services between three distinct customers: buyers ordering products, small businesses & merchants selling goods, and the community of Postmates physically connecting each leg of the exchange.
Managing a multi-faceted marketplace also means managing the risk of fraud throughout the entire ecosystem. Consumer facing services confront fraudulent behavior of all kinds — but on-demand platforms are uniquely exposed to fraud because we have to assess risk in real-time. And effectively doing that means using predictive models to evaluate risk in a scalable way. But while some might think the solution begins and ends with machine learning, the approach we’ve taken at Postmates is based on a fully integrated team; that includes world class operations, analytics, product, and engineering groups — working together with data science to make sure you can access everything that’s available across your city and fraudsters can’t.
Creating Data (And Making it Mean Something)
Before data scientists can predict the future, data engineers need to paint them a picture of the past. From tracking events to database construction, getting data and making it available to data scientists has to come first. The strength of a model, to predict or anticipate outcomes, depends on the data you have to feed into the model in the first place. But even after data production, there’s scope for operations specialists to clean, curate, and refine what the data engineers provide.
Within Postmates, our Risk Operations team hand labels fraudulent transactions to help us differentiate between different types of chargebacks (e.g. account takeover, customer service issues, etc). This allows us to reduce noise and build models that are surgically trained to target specific patterns of fraud. Operations is an often overlooked component of what’s frequently considered a “data science” problem, but there’s immense value in having a team of experts curate data to build higher quality training sets.
Data Science Predicts the Future, Product Decides How to Change It
Building a product or platform that stops fraud is a bit like designing a pasta strainer. You want features that let good users pass through while keeping bad users out of the sink. In practice, this means intentionally inserting points of friction like mandatory credit card scanning or similar verification steps. While the particulars will always depend on the order-flow unique to each product and platform, the goal should always be to intervene in ways that stop fraudsters from placing orders without ever making good users feel like anything is unusual or wrong.
Another product approach is asynchronous evaluation where we make a decision at one point in the order flow but don’t take action until sometime later on. For example, we might make the decision to require more information from a customer when they add their credit card, but wait to actually intervene until later on at check out. This approach is often taken to allow computationally expensive models time to run without forcing good users to sit through what feels like lag, but from a product perspective it also has the benefit of making it harder for fraudsters to adapt to our defenses.
Analysts != Data Scientists
While engineering, operations, product, and data science are core parts of the team, it’s also important to note the role that analysts play. A risk analyst should quite literally know everything about everything, and that starts with not restricting their role to data science lite. We believe analysts should have a wide range of responsibilities including reporting on key metrics, evaluating product changes, assessing 3rd party vendors, and, yes, assisting with the data science workflow. But where a data scientist maintains a narrow focus on model-building, an analyst should have a bird’s eye view of everything your team is doing and be the person on point for running the numbers and knowing what they mean.
Assembling a fully integrated team is key to addressing fraud — and many other data science heavy problems like it. It takes a diverse set of skills to deliver on any sensible risk mitigation strategy and that means appreciating everything that comes before, goes after, and happens alongside the data science. That’s how we’ve built out our team here at Postmates and for those of you just starting to put together your company’s financial immune system, hopefully this will help you accomplish the same.
