My love affair with BigQuery, DoiT, and superQuery
I am a data nerd. I have vigorously resisted upgrading to the “data scientist” title that has become popular recently. I’m sticking with the title the jocks gave me back when I was high school, “DATA NERD.”
The beginning of my love affair: BigQuery
As a data nerd, I’m generally the guy looking at new data tools a few years before they become popular. I was a very early adopter of Google BigQuery. I was using BigQuery in the bad old days when BigQuery only supported “Legacy SQL” and I had to write disgusting nested queries to do anything sophisticated. Why did I fall in love with BigQuery?
- It was ridiculously fast
- It was super cheap (in fact, it started out free!)
- I had a feeling Google was going to turn this into something amazing
I was right. Google did turn BigQuery into something amazing. I have been part of every major alpha and beta release in BigQuery over the last seven years. The developments have been incredible. Some of the highlights that I love:
- Native support/integration to Google Sheets
- Views
- Scheduled queries
- Window functions
- Partitioning
Great work BigQuery team! You have built a technology that is best-in-class (but you still owe me the native connection to PostGres that you promised last year!). I don’t think Amazon will ever be able to catch up, but there is a big problem. Most organizations only have a few data nerds. I can sit and do research using any tool I want, but when I want to start putting things into business applications, I need to align with the rest of my technology teams. The Enterprise Architects, DevOps, and Data Security teams need to all feel comfortable that BigQuery and the surrounding Google Cloud Platform (GCP) provides reliability, security, and documentation to track ongoing changes and upgrades.
I wasn’t able to provide the comfort that my team needed, and they couldn’t get past the “Google is for start-ups and Amazon is for enterprise” mentality. GCP documentation is notoriously unclear (or non-existent), changes and releases frequently occur with no announcement, and Google support is very hard to reach.
DoiT to the Rescue
A colleague referred us to DoiT. DoiT is a company that provides advanced technical support for companies working on the cloud. We started working with DoiT when we were a small company with only 13 employees in 2016. I remember vividly a Zoom call (we were early adopters of Zoom also) with one of DoiT’s founders, Vadim Solovey. Our CTO, Chief Architect, and lead developers bombarded Vadim with tough questions. Vadim plowed through all of the questions with grace. Vadim seemed to be the most knowledgeable cloud expert in the entire universe. My team was convinced also. GCP might be tough to work with, but if we had support from DoiT, we could probably get through all of the issues. We were right. Today we’re a much larger company, and we have built a lot of world-class technology. We could not have gotten to this stage and avoided pitfalls without ongoing support from DoiT.
After my first meeting with Vadim, I remember thinking to myself that if Vadim can pull together a team of people with even 30% of his knowledge, he could build a pretty amazing company. It seems like I was right. DoiT has grown into an international company and raised a large equity round last year. All of their success is well deserved.
Taking Things to the Next Level: superQuery
Last month DoiT bought a company called superQuery. Before the announcement, I had been playing around with superQuery’s Chrome extension for a few weeks. I’m still a superQuery newby, but it seems like a very powerful tool for data visualization. Also, superQuery added a lot of basic features that were missing in the BigQuery Web UI like tabs and cost visualization. superQuery also resolved a critical issue in the new BigQuery Web UI to enable freezing of the header row. Nice!
DoiT, thanks for all of your help over the past few years. I hope you guys have success with superQuery, and I’m looking forward to seeing more great developments in the future.
I’ll love you guys forever!
