Introduction Earlier this week, I gave my first external presentation at Data Teams Summit. I know everyone here has given a presentation at some point, but it’s always important to reflect on how to approach a presentation so you leave a lasting impression. Storytime Just like an essay or even a blog…

Introduction One of the first blogs I wrote back in the days of being a Medium noob (I’m just an advanced noob now) was about S3 lifecycle policies. In recent weeks, I’ve been playing around with these more due to our never-ending pursuit of Cloud cost optimization, so I thought it…

Introduction I’ve previously written about the merits of instance fleets with EMR clusters. …

Introduction Conference/virtual conference 2023 edition kicked off yesterday with the State of Data Conference, which was hosted by Ben Rogojan on his Seattle Data Guy YouTube channel. Here were my main takeaways from a great group of speakers and an overall strong coverage of content. Self-Service Analytics For those who want to watch…

Introduction While my team has been heavy users of Databricks for close to a year now, one area where we’ve been lacking is with proper CI/CD practices when it comes to our jobs. …

Introduction I’ve written quite a few posts on Databricks in the past, but the best practices post I made back in August was premature. …

Introduction I’ve mentioned before that I’m a lurker on Reddit’s wonderful data engineering community. I usually just read, but there was a thread yesterday that I couldn’t resist commenting on, dealing with cutting Cloud costs in data engineering work. …

Introduction Is it time for another Databricks-focused post? Anyways, I’ve come across some various new best practices (including a few centered around cost-savings), and wanted to share those so others could take advantage of them if they weren’t already. SLAs When all of our jobs were previously run through Airflow, we discussed…

Introduction Typically, the initiatives our team has worked in the past are either enhancements to already-existing features or completely new datasets that are separate from our “core” data that we’re responsible for. …