How to use a Makefile to speed up your dbt project workflow

Reduce command fatigue and group related commands for easy reuse

Build, run, test, docs, snapshot, clone etc. There are a lot of dbt commands to run and, if you’re like me, you’re constantly cycling through your command history trying to find the one you need — Just to avoid having to type the whole thing out again. It’s especially true when you often perform multiple commands together. There must be a better solution?

Catcher in the BI

Makefile to the rescue

Unless you’ve compiled software, you may not have come across a makefile before. It’s used to manage all the dependencies and commands needed to compile code, but you can also use them to automate any tasks in Linux or Mac OS.

With a makefile, you can basically create your own abstracted commands. You might need this if:

• You run the same commands often
• They are long commands and you don’t want to type them, or search your command history
• You want to logically && group commands together
• You want to share commands, or commit common command patterns into your project

We don’t need to dive too deep into the finer details of makefiles. The main thing we need to know is that we can create rules, which consist of targets and commands. We’ll be using a special target called a ‘Phony’ (more on that below) that allows you to specify commands to run.

A simple example

Here’s a simple makefile that will generate and serve dbt docs:

# Generate and serve docs for default target
.PHONY: docs
docs:
	dbt docs generate && dbt docs serve

1. In the root of your project, create a file with the name Makefile (no file extension) with the above lines.
2. From the root of your project, run make docs and you’ll see the command dbt docs generate && dbt docs serve is executed:

$ make docs
dbt docs generate && dbt docs serve
04:35:12  Running with dbt=1.7.9
04:35:12  Registered adapter: duckdb=1.7.2
04:35:12  Found 7 models, 25 tests, 3 sources, 0 exposures, 0 metrics, 582 macros, 0 groups, 0 semantic models
...
...

.PHONY

You might have noticed the line starting .PHONY, this has nothing to do with Holden Caulfield, it’s a way to define custom targets, called ‘Phony targets’. These are makefile rules that reference commands and not files.

All your need to remember is to identify your targets as ‘.Phony’ for them to work properly. You can do this above each target like in the simple example above or, alternatively, declare them all in one place, like this:

.PHONY: build test docs help

This defines four Phony targets: build, test, docs, and help; and placed at the top of your makefile.

Build on it

The commands that you add really depends on your personal workflow, and that’s the beauty of it — They’re your specific set of useful commands for your project.

Here’s a few more basic examples that you might add:

.PHONY run seed docs kitchen-sink clean 

run:
 dbt run && dbt test
 
seed:
 dbt seed && dbt run
 
docs:
 dbt docs generate && dbt docs serve

kitchen-sink
 dbt clean && dbt deps && dbt seed && dbt run && dbt test

clean:
 dbt clean && dbt deps

Advanced examples

You can take it up a level by passing variables from the command line to your makefile, and even loading environment variables.

Passing variables from CLI

Let’s say you wanted to have a dbt build command and be able to specify your dbt environment target:

.PHONY: build

build:
 @echo "Building project with environment: $(TARGET)"
 dbt build --target $(TARGET)

Then you can specify the dbt target to use when running the make command:

$ make build TARGET=dev
Building project with environment: dev
dbt build --target dev
04:11:42  Running with dbt=1.7.9
...

When used together with environment variables, you can see how a makefile can become a really powerful tool for manipulating your project.

Load environment variables

To load your .env file, add this to the top of your makefile

include .env
export

Your makefile will now load in all our .env variables. Maybe you specify the default dbt target, for example:

TARGET=dev

Then, update your makefile to check if the .env variable is set:

include .env
export

# Default value for TARGET if not set
TARGET ?= $(TARGET)

.PHONY: build

build:
 @echo "Building project with environment: $(TARGET)"
 dbt build --target $(TARGET)

Now, when you run make build , the target defined in the .env file will be used, which is dev. But you can override it by specifying the TARGET when you run the make command:

# override the default .env target
make build. TARGET=prod

This turns makefiles into sort of bash scripts (Linux pros will probably kill me for saying that), especially when you consider that you can also load environment variables.

Help section

When you’re done with your commands, add a help section that will explain all of the available commands that you’ve created. This really makes it almost like your own custom CLI tool!

# Show available commands
.PHONY: help
help:
	@echo "Available commands:"
	@echo "  docs - Generate and serve docs for default target"

Boilerplate makefile for common dbt tasks

Here’s a boilerplate that you can use to get started. Nothing fancy, but it’ll get you going:

Remember to create a .env, if you haven’t already, file with the default TARGET and PROFILE for your project.

Makefile in action

To give you an idea of what a makefile like look like in an actual project with specific requirements and dependencies, check out what Alex did in his Zero to dbt project’s makefile:

include .env
export

PHONY: run clean clean_log duck_dev duck_prod

DBT_TARGET = dev

run:
 @echo "SPODBTIFY_SOURCE_PARQUET = $$SPODBTIFY_SOURCE_PARQUET"
 dbt run --target $(DBT_TARGET)

doc: 
 dbt docs generate && dbt docs serve 

duck_dev:
 duckdb spodbtify.duckdb -cmd "USE spodbtify.dev; show all tables"

duck_prod:
 duckdb spodbtify.duckdb -cmd "USE spodbtify.prod; show all tables"

clean: 
 unset SPODBTIFY_SOURCE_PARQUET && dbt clean && rm -rf *.duckdb

clean_log:
 rm -rf logs/*.log

This project uses duckdb, so Alex has made some targets to run duckdb and show the tables from prod and dev. So, you can see, the way you use a makefile in your project will heavily depend on your configuration.

Conclusion

Makefiles can be a really powerful way to own the commands that you use. If you’re often running a bunch of commands, or have project-specific common tasks, it can be a life saver.

One possible downside is by abstracting the original commands, you distance yourself from those commands. You might forget the original commands, or miss out on some new feature or usage pattern. So it pays to go back and review your makefile often.

Are you using a makefile in your project? Got any tips or cool usage examples? Please share in the comments.