BigQuery Editions become Generally Available!
Hopefully, you’ve not been living under a rock for the past few months since Google publicly announced what is probably the biggest change to the BigQuery billing model since the introduction of BigQuery itself, but just in case, here’s some more information on it.
What are BigQuery Editions?
Before we cover Editions, I think it’s worth a little history lesson. Let’s take a look at a few key moments in BigQuery’s past.
In the beginning there was “On-Demand”
This was the default BigQuery billing model. If you don’t know what billing model you’re using, then you’re almost certainly using the “On-Demand” model.
With the On-Demand model, you pay based on the volume of data being scanned by your queries, which has a fairly generous free tier that includes 1TB of data per month, per project. For example, you could query one of Google's Public BigQuery Datasets. Let’s take a look:
SELECT *
FROM `bigquery-public-data.wikipedia.pageviews_2022`
WHERE TIMESTAMP_TRUNC(datehour, DAY) = TIMESTAMP("2022-06-30")When we run this query, BigQuery will analyse 5.65GB of data (which is helpfully shown in the console:
This shows even before we’ve run the query, so we’re able to calculate the cost of a query before we’ve even run it! Assuming we’ve used up the 1TB free tier, then we can see the query will cost $0.035 (3 and a half cents) based on: (5.65/1000)*6.25.
On-Demand provides the best end-user experience by far. Queries are submitted and complete in the fastest time possible based on the available resources from a shared pool across the region you run the query. Nor are there any limitations in terms of features. Want to use BQML? Sure. How about Customer Managed Encryption Keys? Go for it.
The only downside is that it’s difficult to keep an eye on the costs, given the free-reign nature of On-Demand, it’s pretty easy to rack up a large BigQuery bill.
Then came Slots…
“What’s a Slot?” I hear you cry! Luckily, the documentation has a really good introduction to Slots. But the TL;DR is that Slots are the unit of compute (which comprises of CPU, RAM & Network) used when processing data in BigQuery.
Slots are essentially a different take on paying for BigQuery consumption. Instead of paying based on the volume of data being processed, you instead pay for the amount of resources being consumed by your workloads — this is also known as “Flat Rate” pricing. Under this model, you would choose a number of Slots to reserve and you would have, and be paying for, this dedicated capacity.
The drawback with this model is that you have had to specify the amount of capacity you want to be available. If your workloads required more capacity than you have available (as all slots were shared across all your workloads within a reservation), then you’d simply have to wait longer for your job to complete. This may be fine for automated workloads, but not the best end-user experience for ad-hoc analysis.
However, under this model, it was easier to control and predict your costs — given that you purchased a specific amount of capacity that was constant. Depending on your scale, this model may also have worked out cheaper than On-Demand if you had a lot of workloads being run concurrently. You could also commit to using these resources, on a monthly or annual basis, in exchange for a slight discount in price. Although given the minimum you could commit to was 100, your costs would start at $2,125/month (if committing annually). It’s worth noting at this point, Flat Rate commitments have been sunsetted and are no longer for sale on Google Cloud.
And now, Editions!
This is where it gets interesting! Google introduced BigQuery Editions back in late-March 2023. Editions are the latest way in which you can pay for your BigQuery consumption.
This introduction carves up the features available in BigQuery into a tiered product model starting with “Standard”, moving up to “Enterprise” and finishing with “Enterprise Plus”. As you move up the tiers, the number of features available increases. Want to use BQML? Then you’d need Enterprise. Need to use Customer Managed Encryption Keys? It’s Plus. However, if you’re fine with one less 9 in availability (aka Service Level Objective), no Data Governance capabilities, and cold-starts for queries, then sure, go with Standard.
Editions utilises the Slot based capacity model as per Flat Rate with one key difference — Autoscaling. In the standard Edition, BigQuery starts off with zero slots provisioned and scales up in multiples of at least 100 once the query planner has figured out the capacity requirements of the job (the cold start mentioned previously). With Autoscaling, you simply set a slots ceiling (the maximum number of slots that can be provisioned) and BigQuery will never go beyond that. There’s a 1,600 slot maximum with Standard. With Enterprise & Plus, you get to set a “baseline”, which is the number of slots that are available (and paid for) 24/7, but you can also set a ceiling with no limit to what that number is (up to the quota).
Like with Flat Rate, you can commit to a specific amount of capacity in exchange for a pretty generous discount (20% with a 1 year commitment, and 40% for a 3 year one).
How do I know which pricing model is right for me?
Good question! The answer can be a tricky one to figure out and is worth a post dedicated to such a subject. Look out for future posts on how to understand your BigQuery costs, and what to consider when optimising!
We can’t talk about BigQuery Editions, without mentioning Compressed Storage! Have a read of this follow up to see what’s changed
Speak soon!
About CTS
CTS is the largest dedicated Google Cloud practice in Europe and one of the world’s leading Google Cloud experts, winning 2020 Google Partner of the Year Awards for both Workspace and GCP.
We offer a unique full stack Google Cloud solution for businesses, encompassing cloud migration and infrastructure modernisation. Our data practice focuses on analysis and visualisation, providing industry specific solutions for; Retail, Financial Services, Media and Entertainment.
We’re building talented teams ready to change the world using Google technologies. So if you’re passionate, curious and keen to get stuck in — take a look at our Careers Page and join us for the ride!
