The Death of BI Tools, and the Birth of Full Stack Data Scientists
I HATE BUSINESS INTELLIGENCE (BI) SOFTWARE!
TABLEAU, LOOKER, COGNOS, etc. I HATE ALL OF YOU!
Wow, I’ve been waiting 20 years to get that off of my chest. It feels great to declare that publicly. Dear reader, go for it! You’re in a home office after all. Close the door, stand up, and scream the truth that burns in your heart.
I HATE BI TOOLS!!!
If we all hate BI Tools, why do we spend so much on these tools? Why did Salesforce spend $15.7B to buy Tableau, and why did Google spend $2.6B to buy Looker? Let’s go back to the beginning.
Data Scientist-Developer Deathtrap
BI Tools were created to resolve what I call the Data Scientist-Developer Deathtrap. In most businesses, executives ask questions and drive strategies that require data-driven answers from middle managers. The middle managers are business-focused, and their questions get passed on to technical employees to get the answers.
The Deathtrap starts because there are really two distinct types of technical employees required to get answers. (A) Developers are required to collect various data from disparate sources and create tools to display data and (B) Data / BI Analysts, which we’ll collectively call “Data Scientists” (as the title is used colloquially today), perform the data analysis. There is a painful see-saw path from Developer to Data Scientist. The middle manager will find his simple question, “what was the average monthly revenue for the email channel,” will take weeks or months to resolve.
Business Intelligence Was Supposed To Save Us
BI Tools targeted frustrated managers. Non-technical managers did not want to wait weeks and months to get answers to simple questions. They reached a peak of frustration with their technical colleagues, and they basically said, “give me the tools and I’ll do the work myself.” That is exactly what every BI tool claims it can do.
Every technical person knows this is complete nonsense. BI tools do not solve the Data Scientist-Developer Deathtrap. They just put a giant bandaid on top of it. The dirty work still needs to get done. Data Scientists and Developers still need to collect data, parse data, and transform data, and that’s why BI Tool integrations can take months to finish.
Why suffer six months on a BI Tool integration? The non-technical manager dreams that the integration will be his last time waiting and suffering in the Deathtrap. He’s looking forward to a magical future moment when he’ll have a user-friendly interface that he can use on his own. It’s a nice dream. I think it lasts about a week. Then the problems start.
- Stuck in Box: On day one, the manager realizes that the BI Tool was not quite as “user friendly” as he had hoped (grrr…why can’t I just move that bar chart to the other side?!?).
- Misinterpretation Trap: Two days into his BI Tool independence, hoping he would never have to talk to that Data Scientist (a.k.a. NERD) again, he finds himself back in the nerd’s office to help make sense of strange results (grr….why don’t the numbers on the pie chart add up to 100%?!?).
- Data is Dynamic: Three days into BI Tool freedom, the manager realizes he is missing a few pieces of critical data, and he’s back in the Deathtrap.
- Data Security Knock Out: On day four, the manager gets a call from the Data Security guys to remind him that he never got approval from them for the BI Tool integration and he can’t use any of the tools to share data with external partners.
If he is lucky, by day five, the manager will get promoted to an executive position for his “brilliant” decision to transition the company to a new BI Tool, and a new sucker will have to inherit his Deathtrap and its multi-million dollar bandaid.
A Better Way — The Birth of Full Stack Data Scientists
After reviewing two dozen BI Tools in extreme depth, my team and I gave up. We made a bold decision. No more bandaids. We wanted to go straight to the root of the problem. We set out to escape the Deathtrap. We took on a mission to transform Data Scientists into Developers. The Data Scientists were scared. They had many concerns, “We just do analysis in Python. We don’t know Git, we can’t write front-end code, we can’t write our own APIs. Yikes!” We told, them, “you’ll learn.” We invested in teaching and training and the investment paid off with huge returns. We created Full Stack Data Scientists.
How is life different with Full Stack Data Scientists? It means there is one address for the manager. He needs an answer, and there is one talented technical employee that builds the API, collects the data, parses the data, performs the analysis, and builds a customized display. It reduced response times from weeks to days or hours. This core capability earned us consistent accolades from partners, investors, and even competitors. Frustrated managers from other companies started coming to us consistently asking for customized tools and dashboards to understand their own data. Today, we’re declaring victory! We finally found a way to help the world escape the Data Scientist-Developer Deathtrap.
COVID-19 further proved to us the strategic value of Full Stack Data Scientists. When COVID-19 hit our target market, Brazil, our data requirements changed faster than they had ever changed, and our underlying data changed also. We needed rapid response, and our Full Stack Data Scientists were up to the task. Within days we had dashboards tracking all of our new metrics, new custom tools to help us identify potential opportunities, and alerts signalling unusual system issues. While our competitors fumbled, we were able to make data-driven decisions to optimize all of our underwriting and collections decisions and consistently produce best-in-class results.
Now is a great time for this revolution. With the backdrop of a tighter employment market, and now from the comfort of their homes, employees are ready to learn new skills. Managers are happy to cut costs by removing expensive BI Tools. It’s time to start building and hiring teams of Full Stack Data Scientists and transform those Nerds into Ninjas.
