SQL is all you need!

Supercharge BigQuery with BigFunctions

SQL is making a comeback as the centerpiece of data platforms.

The SQL Data Stack is the future. It’s not just a buzzword but a transformative approach to how we handle our data operations, making SQL all we need!

I’m Paul, the Head of Data at Nickel. For six years, I’ve been immersed in the world of data, watching our user base on BigQuery grow from a handful to over 200. But more than just managing growth, I’ve been focused on a concept close to my heart: empowering data analysts. Today I want to explain how to do it with a SQL Data Stack.

The Problem with Today’s Data Stack

Let’s face it, the modern data stack, for all its strengths, is complex. We’ve got this plethora of tools around data warehouses. These tools, while powerful, often don’t quite fit our custom needs. And what do we do? We end up with a myriad of custom Python scripts for data ingestion, transformation, and actions. It becomes a complex beast to manage.

Modern Data Stack is complex with myriad of tools and tons of custom python scripts

The Vision: A Single Data Warehouse

Imagine a world where your data warehouse is the only tool you need. A world where you don’t need a multitude of external tools or a patchwork of scripts. This is the essence of the SQL data stack. It’s about leveraging the power of SQL within your data warehouse to do everything — from data ingestion and transformation to activation and beyond.

With SQL Data Stack, the data-warehouse is supercharged and can handle any task.

What is the SQL Data Stack?

The SQL Data Stack is about having a data-warehouse supercharged with functions that can handle any task and that you call in SQL.

1. Load Data from any source → run a SQL query
2. Make Advanced Transforms → run a SQL query
3. Take Actions from data → run a SQL query

With Advanced Functions available from SQL, SQL is all you need.

The Benefits of Embracing SQL Data Stack

Adopting this approach comes with some exciting benefits:

1/ Simplification: Imagine replacing numerous tools and Python scripts with SQL queries calling specialized functions. That’s the simplicity we’re aiming for.

At Nickel we used to have some advanced transforms out of dbt because we could not perform them in SQL. By making them available from SQL, we can have the whole picture in dbt again.

2/ Self-Served Data Analysts: We talk a lot about self-service BI, but what about self-service for data analysts? It’s about giving them a catalog of functions to meet all their needs, right within the data warehouse.

At Nickel, data-platform engineers create once a function callable from SQL to send slack message, and now any data-analyst can use it without asking for any use case: be it to send figures to some teams or to alert when some anomalies are detected in metrics.

3/ Empowerment: At Nickel, our data analysts are fully embedded in teams. They handle everything from ingestion to actions, using tools created by our data platform engineers.

At Nickel, a data-analyst can with a single SQL query: predict the points of sales which are going to be out of stock + directly call internal apis to order more stock for them. This is what we call empowerment.

Introducing BigFunctions: Making SQL Data Stack a Reality

This brings me to BigFunctions, an open-source project I created to extend the concept of self-service data analysis beyond just Nickel. BigFunctions is essentially a framework, much like dbt, but for building a catalog of functions within your data warehouse.

How BigFunctions Works

• BigFunctions is first a framework: Like dbt uses SQL and YAML files for transformations, BigFunctions uses YAML to configure functions.
• CLI for Testing and Deployment: Similar to dbt, it offers a command-line interface to test and deploy these functions.
• Community-Driven: It comes with a growing community of contributors, currently boasting over 120 functions that you can deploy with one command.

BigFunctions is a framework to build a governed catalog of functions

A Practical Example: The Power of Three Functions

Let me illustrate with a simple example. Using BigFunctions, you can:

1. Load Data: Use a function to load any file from the internet directly into your data warehouse.
2. Transform: Apply advanced transformations, like forecasting using the Prophet library, all within SQL.
3. Activate: Send insights or data directly to where it’s needed, like sending a Slack message with sales predictions.

The Advantages Are Clear

• Separation of Responsibilities: Data-platform engineers create the tools, and data analysts use them independently.
• Powered by Community: Over 120 functions and 30 contributors, ready to use.
• Simplicity and Scalability: A streamlined, reliable, and scalable approach to data management.

Join the Movement

If you’re as excited about the SQL data stack as I am, I invite you to explore BigFunctions.

Check out the GitHub repo, contribute, or just add a star (🙏) if you like what you see.

And if you’re keen to dive deeper or have questions, let’s connect on LinkedIn. I’m always eager to discuss how we can push the boundaries of data management and empower our teams.

Together, let’s make the SQL data stack the new way!