If you have ever received a huge csv file that you had to analyze or just quickly wanted to peek into to check it’s structure your go-to tool is usually pandas and a small Python script. But if you are like me and always have DataGrip (or any other JDBC-compatible…

Data contracts help you document and enforce the shape and metadata of your records through data pipelines and processing systems. Their main goal is reducing surprises and getting rid of undocumented changes. For example, if data producers and data consumers agree that the data interchanged between them has a specific…

At work I had the opportunity to start getting familiar with Streamlit for a project so I got motivated to create a side project with it during the weekend. After a bit of research I got to the conclusion that there are not enough examples of building live dashboards with…

Experimenting with new data tools is always fun; Today we’ll take a look at how we can leverage Materialize to build a live map of trending Twitter topics. We’ll be using Redpanda as a fully Kafka-compatible storage layer for our data, from where we’ll build the live and breathing materialized…

It took a long time to shift the mindset of the software industry to be user-first. The rise of a product-focused mentality and product management best practices helped a great deal in pushing this forward. But when the users are developers, all these decades of established processes are suddenly thrown…

This is one of those articles that I wrote to solidify my understanding of something, it’s not meant to be educational (except for myself, hehe), but I find it helps a lot to write things down, and it’s a big plus if someone else finds it useful. Like most people…

One less reason to leave your favorite place, the terminal! Install macOS prerequisite: brew install swig Install ykman with pipx, the official YubiKey manager CLI application pipx install yubikey-manager Usage To list all available accounts ykman oath accounts list Generating an access code for an account that’s called danthelion:daniel@arecadata.com ykman oath accounts…

One of the biggest challenges software engineers — especially beginners — face is learning how to use a new piece of technology. This is to a large extent true for streaming data platforms, as they are considered advanced by nature. In this tutorial, we will create a simple terminal-based chat…

The source I have recently discovered that Wikimedia (The organization behind Wikipedia) has a publicly available streaming API for all Mediawiki sites (wikidata, wikipedia, etc.), called the EventStreams, which uses the Server-Sent Events protocol and is backed by Kafka internally. EventStreams provides access to several different data streams, most notably the recentchange…