Why Migrating your Data Warehouse to Google Big Query can be the Next Big Thing for your Enterprise?

Pluto7 Inc.
Jun 12, 2018 · 3 min read
Design By Abhishek

For most enterprises that have large amounts of data, legacy systems come with certain problems attached. The inability to scale, compute, and store large amount of work to maintain ops of the enterprise data warehouse (EDW), high cost, and the potential for the system to not be able to keep up with forecasted growth, causes what seems like unavoidable headaches. In this article we will discover what BigQuery is and how other enterprises migrated their data.

BigQuery (BQ) is a Google Cloud EDW optimal for analytics. BQ stores petabytes of data at almost unlimited scale and allows for analysis and manipulation with the convenience and speed associated with SQL. BQ is encrypted, durable, and highly available. The system is fully managed and serverless. Many companies are migrating their data into BQ to improve their performance and promote innovation.

Migrating from a legacy EDW to Cloud Storage to BQ, as well as the flow, is made easy with the help of partner tools. One such partner tool is Google Cloud Dataprep which cleans the data, as well as detects and visualizes the schema, distributions, mismatched and missing values using histograms and ranges.

Yahoo Case Study

Yahoo deals with large volumes of data every day. At the beginning of their journey, they decided that speed at scale was most important. One of the reasons Yahoo chose BQ is that there are few options at scale. BQ is fully managed, and Yahoo preferred SQL. BQ has BI integration and allows for independent scaling of storage and compute.

The benefits of migrating: Yahoo needed to migrate from their V1 system to enable all required users to have access. For that purpose, they built an on-prem system. Technically, users had access but the system was too slow and users essentially still didn’t have access to the data they needed. Yahoo has moved multiple petabytes to BQ and now users finally have access to data!

New York Times Case Study

NYTimes needed their data all in one place. At the beginning of their journey with BQ NYTimes was using Amazon Redshift, spent $1 million on Hive based warehouses, and had two Oracle and a standalone warehouse.

The benefits of migrating: NYTimes found that many people were starting to significantly dislike working with data, but after migration occurred, people began having fun again.

“When query results are returned quickly, you truly have the opportunity to innovate.”-Nick Ursa

NYTimes found that the web interface was a huge win because it enabled casual access. They could open up a dataset and someone could to start playing with immediately. There is a whole periphery of SQL users and enthusiasts who can now work without setup.

“The accessibility of joins opens up a lot of doors.”-Nick Ursa

NYTimes has moved on from clickstreams and is now busy replicating the most important datasets it owns. BQ puts print and digital on the same page.

Blue Apron Case Study

Benefits of migrating: Blue Apron shifted its focus away from tuning for performance to building the right data model because BQ’s performance is stable and reliable. Snapshots against date partitions allow them to query against large history tables very cheaply without having to concern themselves with index management. This is a huge time-saver and has lower cognitive overhead. Streaming inserts enable real-time incremental updates for rich event and time series data.

Blue Apron wanted to use Looker which provides a convenient BQ connector. Schema Evolution features add a nice layer of resiliency to their ingestion pipeline. Supports for join is another key feature for Blue Apron. All of this allowed the migration of many four pre-existing data with very little overhead. This gave them the time to re-think their denormalization strategy while immediately gaining the benefits of a much faster query platform. Also, the Dataproc BQ connector gives them the convenience of integration with Spark ML pipelines.

Start the process of migrating your enterprises data over today and reap the benefits of BQ’s scalability and speed. Discover for yourself if BQ is best for your company. Contact us to know more.

Pluto7 Inc.

Written by

Google's preferred Partner In Machine Learning & AI. Machine Learning & AI as a service.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade