How Riskified Turned Investment in Tech Debt Into a Money Generator
As product managers, we face a constant dilemma of what to prioritize first.
On the one hand, we are pressured by sales or customers to deliver the next feature, and on the other hand, by dev teams who feel the backlog is threatening to suffocate them. So should we continue chasing after the next feature or stop and re-visit our existing features, making sure they still deliver what they promised?
This typically relates to product managers in relatively established companies in the growth stage that are getting more traction and traffic but still carrying legacy code from when they were small.
I’d like to share some ideas on how investment in infrastructure could increase your revenue and not slow you down. Use these ideas when you are pitching infrastructure investments to your stakeholders.
How tech debt was created in Riskified
Riskified turned eleven just a few months ago. In the early days, the company was focused on fraud protection in traditional industries. Shipments took seven days to arrive, so a delay of even an hour for the benefit of fraud review would not be noticeable to buyers. Even though Riskified used the latest technical stack at the time, it never had response time as a constraint, so other considerations like cost and code delivery took precedence. As Riskified grew and opened up to new markets, it was clear that a sub-second analysis was required for some industries like food delivery, rideshare, and online gaming.
The dilemma was whether we needed to make a limited change that would give us the sub-second response or go into a full head-on rewrite of our online analysis service.
When answering this question, it was easy to think of the apparent benefits of code refactoring from the tech perspective.
I’d like to suggest some perspective on how the business should look at it.
Where can tech debt investment increase your bottom line?
It’s very common that at some point in a startup’s life cycle, it invests in infrastructure. Although the initiative should come from the dev organization, the business must embrace it and give it the resources it needs. Here are just some of the ways it can pay off.
Downtime or service disruptions
It doesn’t matter what your service is; if it’s experiencing frequent downtime or even slowness, it can cost you a lot. As product people, we like to look at data, measure your backend API response time, and filter on the 99th percentile. Pay special attention to your critical APIs and if they get any timeouts. Try to quantify these disruptions into revenue to see how much bad infrastructure is really costing you.
At Riskified, we performed an in-depth analysis of fraud decisions that were taken with missing data due to timeouts in our internal system. We analyzed how many of these decisions turned out to be wrong and the cost implications.
SLA requirements
As your company expands, it will increasingly engage with larger organizations, often with higher expectations and greater demands. They require higher security and privacy standards and stronger SLA in terms of uptime and response times. Without investing in those areas, even if your sales teams are able to make the sale, it will get stuck in legal trouble for months until dev closes the gaps, meaning lots of lost revenue.
Identify potential pitfalls as early as possible and get a headstart in closing them.
Talk to the sales organization in your company and map sales opportunities lost due to SLA compliance. Create an estimate of the lost potential revenue and use it when making your pitch.
For Riskified, response time was brought up often during sales calls. It was clear that it had to be a focus, but bringing all the numbers together was the real game changer in our decision-making process.
Cost reduction
There are always opportunities to save on costs. Nobody writes efficient code right off the bat; it takes iterations and investigations to find buckets of optimizations. FinOps is already a position you’ll find in many companies; use this resource to find your most costly service and start with it. Raise awareness with your dev teams on the actual costs; you’ll be surprised at how sometimes simple actions can save a lot of money.
As a rule of thumb, you can save at least 15% of infrastructure costs — money on the floor that must be saved.
For Riskified, we moved from a monolith to a modern microservices infrastructure that can be scaled up and down as needed. We removed the usage of traditional, costly relational databases and replaced them with cheaper (and faster) alternatives.
What can you achieve?
Returning to the dilemma at Riskified, we chose to look tech debt in the eye. We realized that the technology we chose to use a decade ago, even though it was the hottest in town at the time, did not fit our future business opportunities.
We re-wrote our order analyzer from scratch, focusing on response time. The first success came a mere four months later, with the first merchant boarding the platform. The new platform ended up being faster and leaner in resources, making it cheaper to run. The new system is more stable and thus maintains a high standard of analysis that improves our fraud decisions and saves us money in chargebacks.
We used this rewrite to add more capabilities for our fraud analysts, and they are now better equipped to train new fraud detection models and set their configurations.
These days, we are working to complete all the feature gaps and migrate the remaining merchants to the new service.
Wrapping up
Tech debt is a common issue that all companies face, but it’s essential to keep it under control so it keeps business goals intact. As a product manager, your role involves recognizing when tech debt requires attention and getting your team on board to address it. To accomplish this, it’s crucial to show how addressing tech debt directly benefits the business. That way, you can guide your organization through the challenges of managing tech debt while staying focused on achieving collective objectives.
