dbt Cloud IDE: Worth it?
With dbt focussing more on dbt Cloud, I examine the Cloud IDE and compare it to its unfashionable older brother, your laptop
Not a Medium Member? Read free here 🚀
Want to see how Orchestra is changing the game by delivering unparalleled cost savings and visibility for data pipelines? Join our rapidly growing Data Communtiy on the Free Tier now.
Introduction
I’ve been using dbt-core about 6 years and my enjoyment for using it for simple data transformation tasks is just as it was back in the day.
All the good stuff is still the same.
- The Schema for the Opportunities Table from Salesforce
- The functionality I use in dbt-core (dbt build, tests, macros, and incremental loading)
- BigQuery’s UI (smh — guys you could have made it slightly better???? Slightly less cluttered? some stuff slightly bigger? no???)
With it being common knowledge that the dbt labs core team are no longer spending their efforts on dbt-core, and focussing on the paid offering (dbt Cloud) the question becomes — why would you buy dbt Cloud?
One oft-touted answer is the dbt IDE / dbt cloud IDE / dbt web IDE / web IDE — sorry — Integrated Development Environment.
In this article I will cover it in very light detail, compare it to how 99% of people develop (including me), locally, and offer some guidelines for choosing what is right for you
Explaining the dbt Cloud Web IDE
Head over to Develop. This takes you to the dbt web IDE:
This is essentially, a browser-based version of VS Code.
You can edit files, push changes to git, and also leverage the CLI // command line interface. What does this mean? Let’s take a look below
In the Cloud IDE, you can:
- Write code
- Test that code by executing the query
- See Lineage (dbt explorer)
- Lint your code by pressing a format button (cool)
- View the Data in the Cloud IDE (the big grey box above I redacted)
- Push changes to Git from the web without downloading anything
These are nice features. Now let’s compare with doing this locally
What about running dbt?
That’s an article for another time and not related to the dbt web IDE! Of course, there are many ways. Running dbt means you need a computer running somewhere. You can pay dbt for this in dbt Cloud.
Adding a fully-fledged Orchestrator like Orchestra to dbt Cloud is incredibly powerful for stitching the end-to-end. If you are dbt-core inclined, Orchestra offers a bring-your-own infrastructure model (see EC2, ECS) for dbt, so you can run dbt in your infrastructure without losing visibility of your data models and dbt tests.
Developing dbt Locally
Developing dbt locally requires you to open up your terminal, pip install dbt
, download VS Code, and also a few other plugins to get the same experience (all free):
- SQLFluff (Linting)
- Turntable for column level lineage (see here)
- vscode-power-user for the dbt explorer lineage bit (and here)
Thanks to whoever curated this, like genuinely. Lots of good stuff in here, well done. https://github.com/Hiflylabs/awesome-dbt
The first thing you need to do is SET UP dbt. This takes a bit of time, but no more than setting up dbt Cloud.
Here is what you get when you’re running dbt locally
As you can see, it looks really like the dbt web IDE! Apart from the command line is in a separate window and it has that retro nostalgic look — sorry one thing I should have added that also hasn’t changed for six years
- How the dbt CLI looks (still love it)
To view data, you’ll probably need to go to your warehouse (BigQuery, Snowflake, Databricks’) UI or you could use Duckdb locally
dbt locally vs dbt Cloud web IDE
Feature differences
Unless there is something new or something I’ve missed, pretty much everything in the IDE you can replicate locally (for free)
Set-up
Installing dbt locally requires you to be able to do a pip install
Ideally you should also know how to do a virtual environment but if you’re new to analytics engineering, even that may not be necessary.
The IDE also requires the same auth config.
Familiarity
If you are one of the few people that write code but don’t use a code editor like VS Code, then you could make an argument for the IDE being more familiar to you and therefore faster…but damn it does look similar to VS Code.
Anything else, Hugo?
One of the great benefits of codifying your SQL is you go from this:
To this:
Version-controlled, governed, well-written, well-structured, codified SQL.
The Developer Environment and its importance is well documented in software. If we are looking to learn from that profession, and extract its best bits for our own, it feels like starting with VSCode is going to be a pretty fucking good move
I forgot the most important one - price!
Developing locally is free! well, as long as you already have a computer. The IDE, you pay for. $100 per user per month. Here’s what you can buy your team for that
- Lunch every day for two weeks every month
- A new iphone for your analysts every year
- A holiday
Ask your team what they’d prefer — the IDE or team lunches. They’ll appreciate being consulted, you’ll grow closer as a group, and they might even enjoy the food.
Conclusion
That’s it! Hope you enjoyed. Hit me up if you’d like to chat data, dbt, or anything else.
Writer’s note: I actually used IntelliJ until 2022 and though vscode has a bit more going on and is a bit harder to get into, it’s so much better
Find out more about Orchestra
Orchestra is a platform for getting the most value out of your data as humanely possible. It’s also a feature-rich orchestration tool, and can solve for multiple use-cases and solutions. Our docs are here, but why not also check out our integrations — we manage these so you can get started with your pipelines instantly. We also have a blog, written by the Orchestra team + guest writers, and some whitepapers for more in-depth reads.
