“Stick Little Thermometers in your Data Journeys” | DataKitchen
Question: What is something the data industry is missing?
I think it’s observability-led DataOps. I’ve come to believe that we, as an industry, will not change how people build things they’ve already made. They’re already being Heroes and have pain, unhappiness, and poor results.
So the idea of Observability first DataOps is to stick a bunch of thermometers all over your data pipelines, models, Vis, and tech stack, then measure all that stuff. Then look at the data and find where the bottlenecks and errors are. Then you get evidence to do the work to fix those problems through automation and testing.
Why Observability first DataOps? Data convinces data people. Otherwise, our experience is that people will continue to “hero out.” They will build these systems where they rush to get something done, are afraid to change it once it’s running, and live with a constant stream of problems from their customers until frustration makes them quit.
Don’t change what you have. Just observe it, get evidence and incrementally improve.
I believe that data can convince our team members to make better decisions. For this approach to work, we need measurements throughout the entire process and an understanding of what’s happening with your data estate as it runs in production. Hence, you know when there are problems or opportunities ahead.
Originally published at https://datakitchen.io on October 27, 2022.
