Why I moved my dbt workloads to GitHub and saved over $65,000
What is dbt Cloud?
From dbt labs, the maker of dbt cloud:
dbt Cloud is a hosted service that helps data analysts and engineers productionize dbt deployments. It comes equipped with turnkey support for scheduling jobs, CI/CD, serving documentation, and monitoring & alerting.
At it’s core it provides the following functionalities
- Job scheduling
- Alerting via email or slack
- Browser based development
Jobs
Jobs allow you to run one or multiple dbt commands. They can be scheduled using basic cron syntax. Job concurrency is limited by plan.
Alerting
dbt Cloud has configurable alerts for jobs when errors or warnings are produced. The alerts can be sent via email or slack.
Browser based development
This feature allows for dbt model development within the browser. No additional tools are needed.
Plans and Pricing
dbt cloud follows the freemium model with 3 plans: Developer, Team, and Enterprise.
Developer
The Developer plan is the best plan as it’s free and provides everything you need for getting started. However, thats about all it’s good for. Unlimited jobs are supported with the caveat that only one can run concurrently. Only 1 project is supported at a time. This may work for smaller teams, or enterprises where all data is stored in an singular database. The real dealbreaker is the one user limit. Most analytics teams have more than 1 engineer. I suppose you could share your passwords with your coworker, but this is not a good security practice.
- 1 user
- 1 project
- 1 concurrent job
- FREE
Team
The Team plan includes everything from the Developer plan with the addition of an additional concurrent job (2 in total) and up to 2 developer seats. No more shared passwords! Unfortunately, we still have the one project limit. Again, this works if you only ever need to manage one database.
- 1 project
- 2 concurrent jobs
- $100 per user/month
Enterprise
The Enterprise plan is where costs start to get interesting (more on that later). It comes with a whole lot of enterprise features, and 10 developer seats. Additionally, there are no limits on projects, and the job concurrency is bumped up to 5. All this for $48,000 per year – billed annually. There is no month-to-month option and if you cancel, the service is not prorated. So for a team of 10, upgrading from Team to Enterprise would jump from $1,000 to $4,000 on a monthly basis just for the privilege of 3 additional concurrent jobs and no project limit.
- 5 concurrent jobs
- unlimited projects
- $48,000 for 10 users
GitHub Workflows and Actions
Jobs
What are GH Workflows?
A workflow is a configurable automated process that will run one or more jobs.
Workflows are automated processes commonly used for CI/CD processes. They can be triggered by actions and events such as PR creation/approval, commit, merge, etc. More importantly, they can be triggered through scheduling.
Alerting
Email and slack integration are both supported.
Browser Based Development
Codespaces brings Visual Studio Code to the the browser. This allows you to bring your entire development workflow into cloud and work from any device that supports a modern browser. VSCode is an award winning IDE that can be customized for any development task through extensions. There is an amazing extension, dbt power user, that gives a more powerful development environment than dbt Cloud — for example, I routinely use python for data science and machine learnings, then switch to dbt for building the data pipeline to support those models.
Plans and Pricing
Comparison
All of GitHub’s plans offer far more job concurrency at a fraction of the price:
- Developer/Free – 20x more concurrency @ $0 savings
- Team – 30x more concurrency @ $96 savings
- Enterprise – 36x more concurrency @ $220 savings
Winner: Github
Browser Based Development
GitHub’s Codespaces is clearly better for more advanced users. For beginners who are just learning, dbt cloud is easier to use.
Winner: Github
Alerts
Both support email and slack integration. However, even with an Enterprise plan, slack integration for dbt Cloud is limited to a single user. This means that only a single user can manage slack alerts for all projects.
Winner: Github
Projects
dbt Cloud plans are limited to a single project unless you have an enterprise plan. GitHub allows you to create as many projects/repositories as you want each with their own actions and workflows.
Winner: Github
Conclusion
It’s a clean sweep for Github. If you are just getting started and are looking to kick the proverbial dbt tires, then by all means, dbt cloud Free is great. However, there really isn’t any reason to not move your dbt workloads to Github actions for any serious analytical work.
