Let’s do better BI (and real ML, while we are at it)

Angshuman Guha
Jul 25, 2017 · 2 min read

We first met as members of an ML (machine learning) infrastructure team at Microsoft. We were working on a myriad machine learning technologies — neural nets, support vector machines, decision trees and forests, scalable linear models etc etc. That work was later subsumed by Azure ML — Microsoft’s cloud-based machine learning service.

We worked together again later, in the analytics division of Sears Holdings. We had an opportunity to learn the needs and problems of a large retail company and the largest loyalty program in the US. We experienced the state of data backends, the inefficiencies of old processes and the difficulties of communication and collaboration in a silo’ed environment.

Three lessons we garnered:

  • It is essential to have a pipeline for getting the data from the fragmented backends (Teradata, Hadoop, etc) into the cloud. All subsequent analytics and ML work then become easier by orders of magnitude.
  • It is important to treat the imported data as the only “truth”, the only source for all subsequent processes to draw from. No more unexplainable inconsistencies in reports and insights.
  • The biggest bang for the buck comes from building end-to-end ML solutions in-house. One-off solutions with a myopic perspective are inefficient in the long run. Some of the powers of ML manifest only when there is a feedback loop from operations to data to insights back to operations.

However, the biggest lesson manifested later. It came gradually at first, but eventually crystallized with the dazzling clarity of the Simpsonian d’oh! With the current hype around ML and a serious shortage of skilled talent, many enterprises find themselves between a rock and a hard place. The state-of-the-art business intelligence tools are non-intuitive, need a ceaseless effort of tinkering by “data analysts” and do not exploit the biggest breakthroughs in Big Data and ML that have happened in the last two decades! While a lot of companies are selling BI tools and an ever larger number are claiming to do ML, there is a market need that remains unsatisfied.

That was the beginning of our journey.

— Vishal & Angshuman (founders of bi++)

Angshuman Guha

Written by

Founder & CEO, bipp inc.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade