datamindedbe

A look at some lesser-known GCP SDK settings and features that make your day-to-day interactions with GCP more enjoyable.

Disclaimer: this is my opinion, not necessarily the one of my employer or any organisation.

An opinionated way to get started quickly

A critical introduction to lakeFS, a new metadata layering solution that brings Git-like operations and versioning to object storage.

Running multiple Airflow instances on the same RDS means a lot of open database connections. RDS proxy can make this setup more efficient.

How to use Cloud Functions and a BigQuery schema generator to make Snowplow bad row schema violation events easily queryable

Using user credentials, service account credentials or the metadata service to obtain access tokens from Google’s identity service

How applications magically authenticate themselves with GCP through their environment, and how to make locally running containers magic too

Introduction to Cross-Origin Resource Sharing (CORS) and the Same-Origin Policy (SOP). Structured as a dialogue, and focused on the why.

The cheapest and simplest way of running computational jobs on Azure is by using Azure Batch. This service enables you to launch managed…

At Data Minded, we are data engineers first and foremost. But in reality, we do a lot of consulting. What is consulting really? I recently…

To go full Kubernetes-native or not?

The world of data is moving and shaking again. Ever since Hadoop came around, people were offloading workloads from their data warehouses…

Last year I blogged about how I got my AWS Pro Architect certificate, which you can read about here…

How to use Snowplow to ingest data from an unsupported source, such as Auth0’s log streaming service.

Adding noise to existing rows, only adding noise to outcomes of tasks performed on that data, or synthetic data generation? An intuition.

As the title says, this article is about how I personally prepared for the aforementioned exams and not about how one should best prepare…

Do you want to run Spark 3 on AKS in pro mode? Meaning no more “just copy-paste the storage account access key into the source code, and…

Now that the Privacy Shield has been invalidated, there are some legal disputes whether European companies and government agencies can…

“It works on my machine”. That’s great. But now how do you make sure it runs in production, repeatedly and reliably? Here we share our…

A high-level comparison of TensorFlow Data Validation, Great Expectations, and Deequ

An overview of cross-domain tracking prevention in the modern browser, specifically of Intelligent Tracking Prevention (ITP) in WebKit.

A short, technical summary of cookie concepts such as “third party cookies” and attributes such as a expiration date, host-only et cetera.

Let’s be blunt here for a second: MSK is not a mature managed service. The author of that post may have changed his mind in the meantime…