After my startup stalled, I started consulting as a way to get back on my feet. One of my challenges was figuring out how to combine my new and improved skills with my data expertise into something useful. Prior to the startup, I had been in Data Visualization (aka Business Intelligence) for over a decade. I’m sort of type cast as a big data expert and didn’t want to go back to pure BI development.
Luckily, my former colleagues at Netflix had been moving in a new and exciting direction. Instead of building insights exclusively with tools like Tableau or MicroStrategy, they had strategically begun deploying custom visualizations. The dev stack consists of Node, React, and a Data Storage layer (Elastic Search, Redshift, Presto etc).
This was an excellent opportunity that combines Web Application development, beautiful UI Design, and big data. Win-win, right?
I jumped at the opportunity right away. Who would miss the chance to be part of an industry changing paradigm shift?
However, I was skeptical. In the early days of the project I thought as most of you are probably thinking:
- It would take longer than Tableau
- It would cost more
- It would be harder to maintain
- It would only be marginally better if at all
Six weeks later and my first project nearly ready to ship, I can say with certainty that Custom Visualizations will be significant trend in Big Data. I’m not saying traditional BI tools will go away. They will still be critical but complemented by custom solutions.
In my estimation there are three major reasons to pursue a custom visualization:
- Performance & Scalability
- Usability and UI
- Licensing Costs
When you’re dealing with petabytes of data, basic in memory cubes and data extracts are not good enough. With custom visualizations you are not tied to a specific storage layer. Everything from Druid and ElasticSearch to Redis become available. You can pre-aggregate to specific levels and store data as simple key-value pairs.
The project that I’m working on has relatively small data. Our primary problem was Usability and UI. Our user base are not technologists and their primary job does not involve becoming data experts. We’ve conducted nearly a dozen focus groups and watched in realtime how much time and effort was wasted searching for the right data. The current BI platform was overkill and seemed detrimental in delivering consistent data across the many locations. We needed something simple, easy to learn, and easy to use.
On average a dashboard targets fewer than a dozen users. However, there are many cases like this project where data needs to be delivered to line of business users. Here we could have thousands of users. And for customer facing reporting, you could have millions of users. UX and UI concerns increase with the number of users.
I could give you a ton of examples of something we can do in custom vs any given BI platform. But, all you have to know is that with a Custom Viz you are not confined to the box of your BI tool. I think you could imagine the implications.
Finally, licensing costs. This one is obvious. The more users you are targeting with well defined requirements, the more you can save with a custom solution. One Viz engineer could easily support thousands of users. Compare the cost of this vs paying licenses on a BI tool. A full stack Viz engineer could cost anywhere between $180K-$300K (big cities), which would still be less than the cost of licenses for just 1000 users:
($420 per year Tableau server * 1000)
+ ( $80K * 1 Tableau Developer)
= $500,000
You know this is a very conservative estimate of how many Tableau developers you need and the cost of salary.
For this price and use case I’d rather get a Viz Engineer and an Analyst/Product Manager.
Breaking Skepticism
Despite the case I laid out above I still had my old school BI frame of mind. This would only be broken through actual experience.
Although this Netflix project took 6 weeks, most of that time was spent figuring out what to build, learning React/Redux, designing an intuitive UI, and conducting focus groups. Actual development was about three weeks. Many of us BI pros have spent as much time or longer building complex Dashboards in other BI tools.
With great platforms like Node and React and great visualization libraries like High Charts, Semiotic (created by Elijah Meeks), and Chart JS, creating beautiful and maintainable custom visualizations have never been easier.
I am now assuaged of the many skeptical ideas I started with. I believe it will be easy to bring on and train my colleagues with Javascript experience to further develop this custom viz. We have access to version control and automation scripts rather than opaque files and manual processes.
In some ways it might be easier than training someone to take over a workbook with dozens of calculated fields, LOD’s, and other hackery.
