Learn how to take your data visualisation skills to the next level. — Okay, let's start with the truth about my background: I’m not a designer, neither a data journalist nor a data scientist. I'm a data engineer, and my daily activities rarely include having to plot charts or conduct any kind of analysis. But even without having lots of experience, I managed…

How Gousto is using CDK to manage custom resources in its data engineering infrastructure — I learned about AWS CDK just a few months after it launched in 2019. At that time, I wanted to build a CDK myself. I didn't know what a CDK was, but my idea was simple: what if I could write Python code to declare the resources I wanted to…

This is how we reduced our data latency from two hours to 15 seconds with Spark Streaming. — We used to get files from the software that controls Gousto’s factory once a day via an SFTP server: several CSV files containing atomic data for each box that went through the production line on the previous day, such as the timestamps of when a box starts and exits the…

Deploying Airflow on AWS is quite a challenge for those who don’t have DevOps experience, that is, almost everyone who works in data. — TL;DR; I created a repo to deploy Airflow on AWS following software engineering best practices. You can go straight there if you don’t feel like reading this post. But I do describe some things you might find useful here. Run a docker-compose command and voíla, you have Airflow running on your…

Sometimes we want to generate fake events to test our pipelines and dashboards. Random events don’t do the job. That’s why I built this Python package. — You want to implement a new streaming pipeline on your workplace and need to show your managers a Proof Of Concept. This POC should allow you to demonstrate some of the functionalities, in this case, generate real-time metrics. However, there is a limitation: you can’t use production data on a…

The lack of content about the productionization of models and scripts. — First, you finish a few online courses. Then study the Iris dataset and solve the Titanic challenge. Finally, schedule some scripts on Windows task scheduler. Anyone who works with data somehow went through those situations. If you got a job in data, soon you realize that you can’t run a…