Easy Monitoring of DBT Cloud jobs with Streamlit

DBT is a core tool the Cazoo data team uses to take raw data from our S3 data lake and transform it into a much more easily digestible form for our reporting & analytics tools (Looker and AWS Athena, to name just two). We run jobs every hour to provide data that’s somewhat near real-time to keep our internal customers informed on how the business is performing.

Using DBT cloud to schedule our jobs comes with one reasonably frustrating issue: detailed information on the status of jobs (including any reasons for failure) is only available for admins, and not standard users. Ideally, we’d prefer to share this information with as wide of an audience as possible without needing to grant admin access. Granting access increases costs, and most people don’t need admin level access; a read-only view on these jobs is more than enough. Fortunately, DBT Cloud provides an API that allows us to pull fairly detailed data from our jobs which is usable. We just need a suitable way to display this.

Below we’ll get into how we solved this, but if you want to jump right into the source code, here it is.

How Streamlit fits in

Streamlit is a fantastic way to build shareable web applications without having any knowledge of front-end technologies I’ve tried similar libraries (like Dash) in the past, but Streamlit has blown us away with how easy it is to use and with how nice the resulting apps look. Streamlit is open source, too, so you can deploy apps locally or on your own infrastructure, if you prefer.

At Cazoo, we have a broad strategy for using external expertise for running tools that lessen the overhead for our engineering teams. Because of this, rather than deploying Streamlit on some Fargate instances at AWS, we’ve participated in the ongoing Streamlit for Teams beta. This allows us to deploy our apps with the click of a button, and let someone else (the nice folks at Streamlit) worry about keeping the lights on.

Besides using Streamlit for the more obvious case of analytics & data science apps, the Data Engineering team is also using it for apps where we want to expose operational information, like dead letter queues or in this case, what’s happening with our DBT jobs).

The DBT Cloud + Streamlit app

Full disclosure: I didn’t build this app. My colleague Oliver Morgans did. I’m just helping out on this one.

We’ve made the app available for anyone who wants to use it, and we’d certainly welcome any questions, comments or pull requests for ways to make this better. I’m sure that once our analysts start using it they’ll be making suggestions for things they want to see . You can find the source code here

Detailed instructions for getting up and running are in the repo README. You’ll need to add a few things like your DBT Cloud account_id and API key into the streamlit secrets.toml, and you’ll probably want to swap out the logo we’ve stored in images, too (unless you’re particularly fond of the Cazoo logo, of course).

We’ve included basic username/password auth in the code. However, if you’re using Streamlit for teams, I’d recommend you use the recently released SSO using Google Accounts as it’s a much nicer experience than typing in a long password every time.

The app is currently composed of a few pieces;

Sidebar

You can filter by all runs, successful runs or failed runs. And you can filter down by project (this project list is driven by a set of project ids you can add in secrets.toml)

RAG Status

We’re pulling the last 500 job runs, and this table will let you sort by the columns to findwhat you’re looking for.

Failed Steps

The step (or steps) that a DBT job failed on is particularly interesting to our analysts, as this often represents a failed unit test . You have the option to select a specific run, and to see any failed steps in that run.

That blue text is a link that will take you to GitHub and the specific file that threw the error. You can configure the base URL for your repo as part of the secrets.toml.

Historical Runs

Picking a job name will also show you the last 10 historical runs. This will also allow you to inspect them for failure steps, if needed.

Wrapping Up

I am a huge fan of Streamlit, and the fact we’re able to easily use it not just for data science but for other operational things engineering-related makes it a key part of our toolbox. Feel free to leave any comments on what you’d add to this app, and please let us know if we got anything wrong — we’re always happy to learn more.