Big Data

Organize your data lake using Lighthouse

Lighthouse is an open source library (using Apache Spark and Scala) that we developed at…

Ever wanted to do better than joins on Apache Spark DataFrames? Now you can!

The new Dataset API has brought a new approach to joins. As opposed to DataFrames, it returns a Tuple of the two classes from the left and right Dataset. The function is defined as

Probably most of you know the basic join types from SQL: left, right, inner and outer. Since these are supported by most of the data-oriented frameworks like pandas, SQL or R, it is easy to restrict oneself to these 4 approaches during the data flow design…

Datagrip is a great database IDE, just like the other IDEs from Jetbrains: Pycharm, IntelliJ, … In this blog, I describe how to connect an AWS Athena database to Datagrip, for my own reference and hopefully helpful for someone else as well. This assumes that you…