Data’s Inferno: 7 Circles of Data Testing Hell with Airflow

Why data testing is hard, but you should…

Automatic semantic versioning using Gitlab merge request labels

Using Scala UDFs in PySpark

TL;DR

In most cases, using Python User Defined Functions (UDFs) in Apache Spark…

Boosting the selection of the most similar entities in large scale datasets

TL;DR

Facilitating Data discovery with Apache Atlas and Amundsen

Apache Airflow HA environment

We have one Hadoop cluster with Apache Airflow as a workflow scheduler and monitor in our current environment. In the near future, we want to build two new Hadoop clusters to handle the production workloads. The airflow instance that we currently use is a single node…

Akka Streams by example: processing files

A look at different facets of Akka’s reactive…

Making sense of Apache Flink state migration with Scala and Avro

In-depth article describing Apache…

Industry2vec: an Implementation for Industry Code Vector Representation

TL;DR or Executive Summary

Evolve your data model in Flink’s state using Avro

TL;DR