Scala

Organize your data lake using Lighthouse

Lighthouse is an open source library (using Apache Spark and Scala) that we developed at…

Ever wanted to do better than joins on Apache Spark DataFrames? Now you can!

The new Dataset API has brought a new approach to joins. As opposed to DataFrames, it returns a Tuple of the two classes from the left and right Dataset. The function is defined as

Probably most of you know the basic join types from SQL: left, right, inner and outer. Since these are supported by most of the data-oriented frameworks like pandas, SQL or R, it is easy to restrict oneself to these 4 approaches during the data flow design…