What It Takes to Create Prefection
Hashmap on Tap Ep. 120
“Very, very few features will ever be billion dollars features, very few products will ever be billion dollar products, but a small number of companies will be billion dollar companies because they successfully solve a problem and capture that opportunity. “
— Jeremiah Lowin, Co-Founder and CEO at Prefect
We recently had Jeremiah Lowin, Co-Founder and CEO at Prefect, on the Hashmap on Tap Podcast to tell us more about what makes Prefect so valuable. They are providing workflow management, orchestration, scheduling, and data flow automation solutions across industries and across client sizes. He is also an advisor to companies such as Spotify and FabFitFun. Notably, he was a PMC member and committer to the Apache Airflow Project for four years as well. Below Jeremiah provides some insights into Prefect and how they developed. Tune in to the full episode to hear more perspectives, their go-to-market strategies, and what they have planned for the future!
Insights from Jeremiah:
A Risky Background
“Most of my career is in some form of data science. The application I focus mostly on is risk management. Particularly, risk management through financial services. I started my career in hedge funds and overseeing risk management through the financial crisis and gained this intense appreciation for a way of making decisions in critical moments which now shows up in Prefect’s product design where we view Prefect as an insurance product more than anything else.”
— Jeremiah Lowin
A Prefect Value
“We are most valuable when something has gone wrong in your world. When everything’s going great in your world, maybe you don’t even really need us. Yes, I’d like to make a real pitch for Prefect as a valuable tool at all times, but the truth is we earn our pay when something’s gone wrong and we deliver value in that context. This idea of risk management insurance has really been a characteristic of my career.
This is one of those ideas that I don’t know if we’ve actually found the right way to slap on our website; like this is your insurance product. I’m not sure it’s a great explicit metaphor but to me, it just drives so much of how we think about the product, its value proposition, how it’s acquired by customers, and even how it’s perceived by customers.
For example when you ask Prefect users or Prefect customers how did this help you? The number one phrase that we hear over and over and over is some variant of “because of Prefect I don’t have to worry about x.” This removal of worry is actually one of the biggest value drivers of Prefect, not our UI, not scheduling, not the proactive features that we deliver, but rather the removal of this concern.
It aligns super nicely with the company’s mission to eliminate negative engineering. Our job as a company, as a product, is not to push onto people a way of doing things but rather to identify the frustrating things that drive them nuts and take them away hopefully through great software.”
— Jeremiah Lowin
Negative Engineering
“The role that insurance plays is the removal of concern and the minimization of the damage of any bad event. Prefect cannot magically make code work that is broken. Sadly, we wish we could, but what we can do is when you run into trouble we can deliver a really great experience whether that’s through debugging, whether that’s through notifications, whether that’s through observability, or whether that’s through just the communication to the person about what happened.
One of the reasons that we call this problem negative engineering is originally we had this idea that negative engineering is the long list of things that you just don’t want to do. Still, you have to do in order to achieve your purpose. Positive engineering, we might say is the manifestation of this problem which is really interesting. It happens between things, between applications, between people, and between groups.
One that I was particularly familiar with was the handoff from a data science team to a data engineering organization. That’s a moment when if something goes wrong it’s not we’re not necessarily sure who’s responsible for the error. We can waste a lot of time figuring out who to point the finger at. Once the error is diagnosed who is the first person to intercept it?
Now if a data engineering team has put this into production they’ll be the person, they’ll be the team that may learn about the error but it may be the data scientist who first built the model or built the analysis who’s the person who good to solve the error. All of a sudden even with the best possible tooling in each group, we have this credit assignment problem between them of how do we get the error, the problem to the person who’s best equipped to provide the solution.”
— Jeremiah Lowin
Gaining Traction and Increasing Productivity
“Again we can see that across handing from one application to another, from a database to a data warehouse, to a pipeline, this idea of how do we get information to where it needs to go is a critical one. A huge part of where Prefect adds value is in minimizing that burden such that instead of having a production incident where you need 30 engineers to roll out of bed and frantically search for the error you have one person who receives one alert, who forwards it to the right person.
That productivity gain that shows up, companies used to quote us these numerical advantages that perfected delivered 99% productivity gains, 97% improvement. Numbers that just didn’t make sense until we realized that it was because it wasn’t just that one person had compressed a week into five minutes. It was that an entire other 19 people didn’t have to show up to deal with this problem at all and were free to work on whatever pursuits were in scope them.
Again just acting as this insurance product giving folks the confidence that when things go wrong they’re going to have the tooling in place has become a primary value driver for Prefect, but people don’t always want insurance and they certainly don’t buy insurance just out of the blue unless they’re an extraordinarily risk-averse type. Do you ask what are people doing when they don’t have Prefect?”
— Jeremiah Lowin
A More Prefect World
“By far the most common thing is they have a homegrown orchestration equivalent because you need to retry some code. What do you do? Do you reach for airflow to spin up the database, a web server, just to retry line code? No, you write a try, accept, and you move on with your day. Then the person next to you has a retry error. You give them whatever little library you rode and pretty soon you’re running an orchestration thing in-house.
We see this over and over and over, especially with our larger customers where they’ve built some tooling in-house. It’s very specific to their use case but it works up until the moment they want to adapt it for a brand new use case and they discover that the primitives they put in place don’t translate. Therefore, one of our product goals is to minimize the cost of acquiring our insurance which is to say in our world the lightness of dropping in Prefect just to target a specific problem that someone’s run into.”
— Jeremiah Lowin
Listen to the episode:
🚨Subscribe to Hashmap on Tap on Spotify, Apple Podcasts, & Google Podcasts.🚨
Related Resources
Ready to Accelerate Your Digital Transformation?
At Hashmap, an NTT DATA Company, we work with our clients to build better, together. We are partnering with companies across a diverse range of industries to solve the toughest data challenges and design and build data products — we can help you shorten time to value!
We offer a range of enablement workshops and assessment services, data modernization and migration services, and consulting service packages for building new data products as part of our service offerings. We would be glad to work through your specific requirements. Connect with us here.
