Dave Flynn·May 18Zero config code review and data profiling tool for dbt projectsPipeRider, the open-source “code review for data” tool just released version 0.25.0 that has an increased focus on dbt. So much so, that all you need to do is install and run it. There’s no longer any need to initialize PipeRider in your project as your data source connection settings…Data Quality3 min readData Quality3 min read
Balu Rama Chandra·May 18Why you need code review for your data projectsAs data engineers, we use code or programming languages such as SQL and Python to process data and create data pipelines. Data pipelines are constantly changing to accommodate business requirements or changes in data sources, and this means your code is also changing In my experience, and I’m sure yours…Data Engineering5 min readData Engineering5 min read
Dave Flynn·May 95 Interesting datasets from the Data Engineering ZoomcampThe DataTalks.Club Data Engineering Zoomcamp recently wrapped up. It was a way for data folks to learn and hone their skills, and provided a series of free modules that take you through the whole process of building a data pipeline, from data ingestion to dashboard. There’s no reason you still…Data Engineering4 min readData Engineering4 min read
Dave Flynn·May 8How to track your dbt project’s BigQuery data warehouse costs (a PoC)Do you want a way to track your BigQuery usage costs? Check out this interesting proof of concept for dbt. Data warehouse usage isn’t getting any cheaper and, as dbt continues to gain in popularity, cloud warehouse usage is on the up, with more and more transformations done in the…Dbt3 min readDbt3 min read
Dave Flynn·Apr 27PipeRider Now Uses dbt Tags to Specify Which Models To ProfileThe way that PipeRider works has changed, especially if you’re initializing a new project from PipeRider v0.23.0 and up. To profile dbt models, you now need to add the PipeRider dbt tag to each model, or otherwise specify which table you want to profile. When running PipeRider in your dbt…Data Quality3 min readData Quality3 min read
Dave Flynn·Mar 28Reduce ‘command fatigue’ working on dbt data projectsThere’s a ton of commands to run each time you update the code for your data project — building models, serving docs, testing, data profiling etc. The whole process can be exhausting, especially if you are building regularly. “As the data stack increases, the number of commands to type increases…Data Engineering3 min readData Engineering3 min read
Balu Rama Chandra·Mar 23How to Catch Common Data Quality Issues That Haunt Data Engineers with PipeRiderDuring my 7 years of experience working in data and analytics engineering, I often ran into data quality issues such as: Duplicate rows Incorrect data types Missing values Schema changes These kind of issues are a common occurrence for data and analytics engineers, and they can cause havoc on a…Data Quality9 min readData Quality9 min read
Dave Flynn·Mar 3How to run dbt with BigQuery in GitHub ActionsGitHub actions, and CI in general, is the perfect tool for data engineers to automate building and testing dbt data projects. To be able to run dbt from GitHub actions, you need to connect to your data warehouse. …Bigquery4 min readBigquery4 min read
Dave Flynn·Feb 24Data Engineering Workshop: Understand the impact of data model changes in dbt with PipeRiderThe Data Engineering Zoomcamp is well underway! It’s a self-paced online introductory course on data engineering that takes you all the way from data ingestion, right up to building dashboards. As part of the week 4 module on Analytics Engineering, we ran a hands-on PipeRider workshop that shows how you…Data Engineering3 min readData Engineering3 min read
Dave Flynn·Jan 19Why you lack confidence merging dbt data model PRsWe love dbt — Its popularity seems to have skyrocketed over the last year or so. Everywhere I look people are writing about learning and using dbt. Applying software development practices to data projects, and the beauty of ELT, has really struck a chord with data engineers. With dbt projects…Dbt4 min readDbt4 min read
