In order to aid in understanding what exactly IS the BigQuery service, here is a quick rundown of what I’d consider the major user-facing components:

• Serverless Service Model
• Opinionated Storage Engine
• Dremel Execution Engine & Standard SQL
• Separation of Storage and Compute through Jupiter Network
• Enterprise-grade Data Sharing
• Public Datasets, Commercial Datasets, Marketing Datasets, and the Free Pricing Tier
• Streaming Ingest
• Batch Ingest
• Federated Query Engine
• UX, CLI, SDK, ODBC/JDBC, API
• Pay-Per-Query AND Flat Rate Pricing
• IAM, Authentication & Audit Logs

There is, of course, much more to BigQuery than this, and we’re taking a customer-centric viewpoint here. I’m ignoring some of the more mundane bits. That said, we previously wrote about what BigQuery looks like under the hood, and the 15 things you probably didn’t know about BigQuery.

Serverless Service Model

Probably the most important aspect of BigQuery is its serverless model (excuse the buzzword). I can say this with a straight face — BigQuery carries some of the highest level of abstraction, manageability, and automation in the industry, freeing you from the tyranny of VMs and CPU/RAM sizing. BigQuery’s compute is incredibly elastic, capable of scaling to dozens of thousands of cores for just a few seconds, while letting you pay only for what you consume; BigQuery is a highly available, durable, and secure out of the box. There, checks all the boxes of being “serverless”, you see…

Opinionated Managed Storage Engine

BigQuery has an amazing storage engine, continuously evolving and optimizing your storage on your behalf — for free and without disruptions.

Colossus is Google’s successor to GFS. Colossus is great — it’s durable, incredibly performant, and super-scalable. BigQuery stores its data in Colossus in the opinionated Capacitor format. BigQuery’s Capacitor does lots of optimizations under the hood, burning a good amount of CPU/RAM in the process (without affecting your query performance.. or your bill!).

One great example of what BigQuery does with storage is automatically re-materialize your data in cases when your tables are powered by too many small files. The “many small files” problem is the bane of existence of a whole generation of DBAs.

Dremel Execution Engine & Standard SQL

Everyone knows that BigQuery runs on top of Dremel. That said, Dremel itself has evolved, and it’s a bit of a different beast these days than what’s described in the paper:

• As of Summer of 2015, 100% of BigQuery users run on a new version of Dremel.
• BigQuery executes its shuffle in-memory in a separate sub-service
• Dremel does stuff like pipelined execution and smart scheduling

Dremel itself is a vast multi-tenant compute cluster. Your queries are just short-term tenants in Dremel, and the Dremel scheduler performs Cirque Du Soleil-like gymnastics to keep all queries running at top shape. Nature of Dremel also makes you immune to any individual node going down — Yay!

Dremel these days supports its legacy SQL-ish dialect, as well as 2011 ANSI Standard SQL dialect.

The Jupiter Network and Separation of Storage & Compute

We’ve ridden this hobby pony to its last legs. Jupiter is Google’s inter-data center network, capable of a Petabit of bisectional traffic, and allowing BigQuery to sling data from storage to compute without a hiccup. It’s the glue.

Enterprise-Grade Data Sharing

BigQuery’s pure separation of storage and compute, coupled with awesomeness of Colossus allows folks to share Exabyte-scale datasets with each other, much like Google Docs and Sheets are shared today.

It’s not an anti-pattern in some architectures to copy data into disparate clusters in order to share the data, creating data silos. I’ll argue that data silos are the worst — you’re playing a game of telephone with data, you’re increasing your complexity of operations, your infrastructure is very inefficient, and in the end your bill is through the roof!

BigQuery doesn’t leverage VMs for its storage layer (not even as accelerators), so you’re going to be immune to any data storage-related shenanigans like race conditions, locking, hot spotting, and you can throw away your knowledge of the CLONE and SWAP commands. Did I mention that Colossus is great?

Finally, serverless nature of BigQuery allows you to share data with other organizations, without ever forcing them to create their own clusters. You pay for storage, and they pay per-query, and it’s all entirely transparent. Who likes paying for idle clusters? That’s right, nobody!

Public Datasets, Commercial Datasets, Marketing Datasets, and FREE TIER!!

Some good folks are leveraging BigQuery’s powerful data sharing features to do some really cool stuff.

Public Datasets Program has the momentum of a downhill runaway freight train. Datasets are added almost weekly, with NOAA data being the latest entrant.

You can either monetize or procure commercial-grade datasets through BigQuery’s Commercial Datasets. Likewise, if you’re using Google Analytics, AdWords, Doubleclick, or Youtube, your data can end up in BigQuery in one click with Marketing Datasets (as folks from Mercedez-Benz, Hearst, and New York Times found out).

Finally, BigQuery has a perpetual free tier, allowing you to store 10GB and query up to 1TB per month.

Streaming and Batch Ingest

BigQuery‘s Streaming API is a rather unique feature. You’re able to stream data into BigQuery to the tune of millions of rows per second, and data is available for analysis almost immediately. This is actually a pretty hard technical problem for analytic databases to solve, so kudos to the team here.

BigQuery’s Batch ingest is no slouch, and there’s a whole separate blog on it (tl;dr: it doesn’t eat at your query capacity, nor does it cost anything!)

Federated Query Engine

If your data resides in Bigtable, GCS, or Google Drive, you’re able to query that data directly from BigQuery without any data movement. This is what we refer to as “federated query”.

UX, CLI, SDK, ODBC/JDBC, API

Typical access patterns, all wrapped around the REST API. One point worth mentioning is how nice it is working with BQ’s semantics:

• Jobs that commit storage (query, load, copy) commit all-or-nothing. There is no need to cleanup failed or half-completed jobs.
• Queries see storage at a snapshot in time. They are thus immune to race conditions, table/row/cell locks, halfway states, whatever.

Pay-Per-Query AND Flat Rate Pricing

BigQuery has two pricing models — the ultra-efficient cloud-native pay-per-query model, and the predictable Enterprise-grade Flat Rate model.

Folks decry the pay-per-query model as being too expensive. I am empathetic to folks having a hard time grok this model, and predict it, but expensive it is not. You ONLY pay for what you consume, and not a penny more, and in analytic workloads (which tend to be volatile) that counts for a lot of dough.

If you’ve a rather large use case and cherish price predictability over efficiency, BigQuery does offer the Flat Rate Pricing model. You pay one flat fee, and all queries are free!

Here’s the cherry on top — you get perfect visibility into either model. You can choose to jump from one model to the next, as it fits your budget needs.

IAM, Authentication and Audit Logs

BigQuery is compliant with Google Cloud’s IAM policies, which allow organizations to carve out high-granularity role and controls for its users.

BigQuery supports two general modes of authentication:

• OAuth (the 3-legged user-involved auth approach)
• Service Accounts (headless through a secrets file)

There are valid use cases for both. OAuth is great if you’re already integrating with Google’s authentication, and Service Accounts work if you’re federating access controls on your side.

Finally, BigQuery’s Audit Logs is a paper trail of all things that happen in BigQuery. A large number of users export Audit Logs back to BigQuery and visualize BigQuery usage in Data Studio in real time!

So there you have it. You’ve made it to the bottom of my tirade. Hopefully I’ve done an okay job detailing the breadth and power behind BigQuery, Google’s analytics workhorse since 2012. Please do leave comments or questions!

The 12 Components of Google BigQuery was originally published in Google Cloud Platform — Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

I agree with the premise that in a fully-transparent and frictionless market, forces of supply and demand meet at the most efficient point, maximizing value for both parties (or at least for the system in aggregate).

However, there’s a number of friction points in play here with Spot Instances:

• Per-hour billing granularity
• High number of instance types prevents commodization of asset down to its raw qualities of CPU, RAM, network, etc.
• Ephemeral non-guaranteed nature of assets
• Volatility of supply and demand
• Volatility of pricing.. averages look great, but highs can be very high.

It’s telling that there is a thriving ecosystem trying to smooth out Spot Instance markets.

You are also operating under the assumption that Preemptible VMs suffer from low availability and have large contention on demand side.

So yes, while Preemptible VMs aren’t hyper-efficient in terms of distributing value, they’re simple and predictable. You know what you’re getting, and our clients tell us that’s pretty important.

This is a fantastic feature of BigQuery, one that BigQuery has had since its General Availability in 2012.

There are some very interesting and unique aspects of BigQuery’s data sharing capability worth pointing out:

• BigQuery Public Datasets is a freight train of innovation and community-building, with Github, Hackernews, NOAA, and many other datasets hosted and updated, and we invite you to partake as well!
• BigQuery is a purely serverless offering. This means that clients do not need to go the extra step of deploying and measuring out their clusters. Reliability is also a nice bonus here, since we abstract away hardware instability and maintenance.
• BigQuery Commercial datasets.
• BigQuery has a perpetual free tier — store 10GB of data for free and query 1TB of data for free per month.
• One-click Integration with Google Analytics, Google AdWords, Google Doubleclick, Youtube, and so on.
• Federated Query and Federated Storage — you can use BigQuery Storage from Hadoop/Spark/Flink/Dataflow, and you can run BigQuery SQL on GCS/Bigtable.

With that in mind, I am here to introduce a new buzzword — Google Data WareCity — it’s much more than a house, you see!

However, do not conflate “custom discounts” with Google’s Committed Use Discount — two very different concepts.

As long as we’re on this topic, I do encourage you to watch the interview with Pivotal at Re:Invent :

https://youtu.be/BUN0wx2e6Dc?t=4m54s

(work on GCP as well)

https://cloud.google.com/storage/docs/storage-classes#multi-regional

The argument in my blog is confined to specific scope — how and why Google’s Compute is more flexible than Amazon’s, especially when it comes to RIs. I am certainly not intending for that argument to be taken as a superset of all arguments around cloud.

Your response does expand the scope of this conversation, and into a topic that Google shines in (and far from accidentally)- powerful higher level services, so I’d love to hear your thoughts on just some of these points:

• Unique to Google — Google has Multi-region Object Storage, in case of those pesky regional storage outages.
• Unique to Google — Bigtable, a high-level serverless NoSQL database, demonstrated to serve 53 million qps by Sungard FIS last year.
• Unique to Google — BigQuery provides the highest level of manageability for an analytics database yet, at Petabyte-scale. Last week Yahoo and NYT detailed their migrations from, well, a different cloud vendor :)
• Unique to Google — Spanner is the only serverless Petabyte-scale strongly consistent regionally replicated horizontal, and highly available RDBMS.
• Unique to Google — Cloud Dataflow is serverless batch and stream processing engine.
• Unique to Google — Dataproc is job-scoped Hadoop and Spark clusters, very compelling to folks coming from EMR, and I wrote on that topic here.

So while I attempted to clarify one bit of debate around just one piece of the puzzle (compute), your chief argument is much broader, and is far from false on Google Cloud. In fact, one may very easily argue that Google is a market leader in offering higher-level services, and I gave just a couple of points of evidence for you to ponder.

I look forward to hearing your thoughts!

Update March 13: Two days later AWS responded to this move by relaxing rules about switching between instance types. However, this does nothing to alleviate restrictions around instance families and generations. You are still limited by network/disk/GPU/CPU characteristics of the instance family.

I’ve been watching discussions around Google’s freshly minted Committed Use Discounts program , and how it compares with Amazon’s Reserved Instances (RIs). The verdict is in — third parties like Rightscale have already done the math and showed Google to be at least 35% cheaper.

It’s easy to overlook the larger impact, and some press has already concluded that it’s an easy glitch to fix (just drop that 35% gap to 0%, you see…). Not so fast. Google Committed Use Discounts are much more than just a “pricing schema”. There are some serious practical benefits that have nothing to do with cost, and the other cloud vendors aren’t in a position to compete here technologically any time soon. In the end, all customers win!

Let’s discuss the what, the how, and the why…

Google Committed Use Discounts vs Amazon RIs

In short, Google now lets users pre-purchase chunks of CPU and RAM on 1- and 3-year commitments in return for substantial discounts, up to 57%. With Google you can create Custom VMs, picking your own CPU and RAM configuration. All instances get fantastic networking, and all instances can get top-notch disk and GPUs. So you are truly buying CPU and RAM, while retaining architectural flexibility. Users get to turn the IOPS/disk/network/GPUs knobs whenever they want, invariant of “instance family” or some other arbitrary (to us) limitations.

This is in sharp contrast to what Amazon offers with RIs — pre-purchasing instance types, which have specific characteristics like “nice network instances” or “GPU instances” or “great storage instances”. Thus with Amazon you’re pre-purchasing a pre-set configuration of CPU/RAM/IOPS/Network/GPU/Disk characteristics with only minimal flexibility (mostly around EBS and instance sizes), and your mobility to other pre-set configurations is severely limited. So you better be damn sure you made the right choice, because you’re living with it for 1–3 years.

Here’s another way to look at AWS instances. A “compute optimized” instance is just another name for an instance with inferior disk, network, and a low amount of RAM. Why not have great everything!!

How how how is this even possible?

Google is able to offer this due to the unique nature of Google Cloud. Google Compute Engine under the hood is NOT a service that sells a bunch of VMs running on specific hardware. Compute Engine is an opinionated, living and breathing supercomputer, continuously carving out resources for its clients in the most optimal fashion (compare this to Microsoft’s perplexing claims in this space) . Complexity is abstracted away, and users are exposed to familiar IaaS primitives — VMs, networking, disk, etc.

Since 2013 Google’s been heavily leveraging Live Migration to help make these primitives as customer-friendly as possible (not to mention, to patch critical hypervisor flaws or to perform maintenance or remove noisy-neighbor problems). Goodbye maintenance windows!

Live Migration also lets us truly maximize performance, and to make that performance stable and predictable. Test us, I dare you! As far as I know, no other cloud has Live Migration, certainly not to the same degree.

Here are some more critical technicals of Google Cloud:

• Google runs homogeneous hardware footprints, manufacturing and designing components that go into its data centers.
• Google’s Jupiter network offers a Petabit of bisectional network bandwidth within each data center cell. Bandwidth does grow on trees, and a major reason why Google’s able to offer juicy services like BigQuery and Spanner.
• Borg is Google’s orchestrator (and a predecessor to Kubernetes), spinning up workloads on-demand, bin-packing resources, performing rolling upgrades, and making sure performance is top-notch all around.

Google’s the only cloud provider running this way, and has been doing it so for the past 3.5+ years. The fact that no other competitor has emerged there in that long of a timeframe is indicative of just how damn hard the problem is, and how many technical barriers there are to creating this offer.

Benefits of Committed Use Discounts

Let’s quickly run through some of the benefits of Google Committed Use Discounts:

• They’re quite inexpensive. I suspect this matters for clients, but don’t hold me to it.
• You aren’t required to pay upfront to get the inexpensive price.
• On Google, you aren’t stuck on an “instance family” and have to creatively compel your sales rep to get moved up to the newest generation. Sadly, your sales rep’s incentives are in direct conflict with yours here, as even Pivotal found out (go watch that video!).
• You aren’t stuck on a CPU/RAM combination that may be inefficient or wasteful for you. Re-carve it using Custom VMs.
• Your network/disk/IOPS/GPU knobs aren’t soldered shut. You retain 100% of the flexibility here. This is a big deal!
• As Urs mentioned in his talk yesterday, you can retire your “Ministry of RI Optimizations”. Stop playing RI tetris, seriously — the sadomasochism is entirely optional.

How do you compete with this?

You can answer Google’s Committed Use Discounts by lowering prices or making RIs more user friendly and less restrictive, but in order to provide Google Committed Use Discounts, you need to do some serious engineering homework:

• Get yourself a custom-manufactured data center stack with very few vendor dependencies
• Make sure this stack is as homogeneous as possible
• Build a Jupiter-like network that lets every instance in a data center talk to every other instance at 10G.. All at the same time..
• Acquire Borg or a similarly-minder orchestrator, and heavily invest in this orchestrator for 10+ years
• Create VMs that give you best-in-class disk AND network AND GPUs, invariant of “instance family” or type.
• Productize Live Migration and give it 3+ years to mature.
• Productize Custom Machine Types

Do not discount (no pun intended) the technical complexity here. These problems are very very hard. As Eric Schmidt has said, Google’s poured $30 billion dollars over the past three years on this bonfire, and it shows. In the end, users win!

So think of Google next time you’re trying to make sense of your cloud bill, or next time your sales rep calls you to re-up your commitment, or next time you’re trying to get a discount from your cloud vendor. You deserve the best!

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!

Why Google’s Answer to AWS Reserved Instances is a Big Deal was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.

Jobs-first Hadoop+Spark, not Clusters-first

Typical mode of operation of Hadoop — on premise or in cloud — require you deploy a cluster, and then you proceed to fill up said cluster with jobs, be it MapReduce jobs, Hive queries, SparkSQL, etc. Pretty straightforward stuff.

Services like Amazon EMR go a step further and let you run ephemeral clusters, enabled by separation of storage and compute through EMRFS and S3. This means that you can discard your cluster while keeping state on S3 after the workload is completed.

Google Cloud Platform has two critical differentiating characteristics:

• Per-minute billing (Azure has this as well)
• Very fast VM boot up times

When your clusters start in well under 90 seconds (under 60 seconds is not unusual), and when you do not have to worry about wasting that hard-earned cash on your cloud provider’s pricing inefficiencies, you can flip this cluster->jobs equation on its head. You start with a job, and you acquire a cluster as a step in job execution.

If you have a MapReduce job, as long as you’re okay with paying the 60 second initial boot-up tax, rather than submitting the job to an already-deployed cluster, you submit the job to Dataproc, which creates a cluster on your behalf on-demand. A cluster is now a means to an end for job execution.

Again, this is only possible with Google Dataproc, only because of:

• high granularity of billing (per-minute)
• very low tax on initial boot-up times
• separation of storage and compute (and ditching HDFS as primary store).

Operational and economic benefits are obvious and easily realized:

• Resource segregation though tenancy segregation avoids non-obvious bottlenecks and resource contention between jobs.
• Simplicity of management — no need to actually manage the cluster or resource allocation and priorities through things like YARN resource manager. Your dev/stage/prod workloads are now intrinsically separate — and what a pain that is to resolve and manage elsewhere!
• Simplicity of pricing — no need to worry about rounding up to nearest hour.
• Simplicity of cluster sizing — to get the job done faster, simply ask Dataproc to deploy more resources for the job. When you pay per-minute, you can start thinking in terms of VM-minutes.
• Simplicity of troubleshooting — resources are isolated, so you can’t blame the other tenants on your problems.

I’m sure I’m forgetting others. Feel free to leave a comment here to add color. Best response gets a collectors’ edition Google Cloud Android figurine!

Dataproc is as close as you can get to serverless and cloud-native pay-per-job with VM-based architectures — across the entire cloud space. There’s nothing even close to it in that regard.

Dataproc does have a 10-minute minimum for pricing. Add the sub-90 second cluster creation timer, and you rule out many relatively lightweight ad-hoc workloads. In other words, this works for big serious batch jobs, not ad-hoc SQL queries that you want to run in under 10 seconds. I write on this topic here.(do let us know if you have a compelling use case that leaves you asking for less than a 10-minute minimum).

The rest of the Dataproc goodies

Google Cloud doesn’t stop there. There’s a few other benefits of Dataproc that truly make your life easier and your pockets fuller:

• Custom VMs — if you know the typical resource utilization profile of your job in terms of CPU/RAM, you can tailor-make your own instances with that CPU/RAM profile. This is really really cool, you guys.
• Preemptible VMs — I wrote on this topic recently. Google’s alternative to Spot instances is just great. Flat 80% off, and Dataproc is smart enough to repair your jobs in case instances go away. I beat this topic to death in the blog post, and in my biased opinion it’s worth a read on its own.
• Best pricing in town. Google Compute Engine is the industry price leader for comparably-sized VMs. In some cases, up t0 40% less than EC2.
• Gobs of ephemeral capacity — Yes, you can run your Spark jobs on thousands of Preemptible VMs, and we won’t make you sign a big commitment, as this gentleman found out (TL;DR: running 25,000 Preemptible VMs) .
• GCS is fast fast fast — When ditching HDFS in favor of object stores, what matters is the overall pipe between storage and instances. Mr. Jornson details performance characteristics of GCS and comparable offerings here.

Dataproc for stateful clusters

Now if you are running a stateful cluster with, say Impala and Hbase on HDFS, Dataproc is a nice offering here too, if for some reason you don’t want to run Bigtable + BigQuery.

If you are after the biggest baddest disk performance on the market, why not go with something that resembles RAM more than SSD in terms of performance — Google’s Local SSD? Mr. Dinesh does a great job comparing Amazon’s and Google’s offerings here. Cliff notes — Local SSD is really, really, really good — really.

Finally, Google’s Sustained Use Discounts automatically rewards folks who run their VMs for longer periods of time, up to 30% off. No contracts and no commitments. And, thank goodness, no managing your Reserved Instance bills.

You win if you use Google’s VMs for short bursts, and you win when you use Google for longer periods.

Economics of Dataproc

We discussed how Google’s VMs are typically much cheaper through Preemptible VMs, Custom VMs, Sustained Use Discounts, and even lower list pricing. Some folks find the difference to be 50% cheaper!

Two things that studying Economics taught me (put down your pitchforks, I also did Math) — the difference between soft and hard sciences, and the ability to tell a story with two-dimensional charts.

Let’s assume a worst-case scenario, in which EMR and Dataproc VM prices are equal. We get this chart, which hopefully requires no explanation:

If you believe our good friend thehftguy’s claims that Google is 50% cheaper (after things like Preemptible VMs, Custom VMs, Sustained Use Discounts, etc), you get this compelling chart:

When you’re dishing out your precious shekels to your cloud provider, think of all this extra blue area that you’re volunteering to pay that’s entirely spurious. This is why many of Dataproc’s customers don’t mind paying egress from their non-Google cloud vendors to GCS!

Summary

Google Cloud has the advantage of a second-comer. Things are simpler, cheaper, and faster. Lower-level services like instances (GCE) and storage (GCS) are more powerful and easier to use. This, in turn, lets higher-level services like Dataproc be more effective:

• Cheaper — per-minute billing, Custom VMs, Preemptible VMs, sustained use discounts, and cheaper VMs list prices.
• Faster — rapid cluster boot-up times, best-in-class object storage, best-in-class networking, and RAM-like performance characteristics of Local SSDs.
• Easier — lots of capacity, less fragmented instance type offerings, VPC-by-default, and images that closely follow Apache releases.

Fundamentally, Dataproc lets you think in terms of jobs, not clusters. You start with a job, and you get a cluster as just another step in job execution. This is a very different mode of thinking, and we feel that you’ll find it compelling.

You don’t have to take my word for it — good folks at O’Reilly had this to say about Dataproc and EMR.

Find me on twitter at @thetinot . Happy to chat further!

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMI family. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!

Why Dataproc — Google’s managed Hadoop and Spark offering is a game changer. was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.