Maher Bhavsar

UDF i.e. User Defined Functions is a very helpful API from Spark SQL. It acts as a column based function.

One of the important case I used it, is for converting complex data structure like arrays and structs to String, the reason being, I wanted to save the data in a csv, and saving in csv wont go through with these data structures. This csv then can be shared to other stakeholders for decision making, so lets checkout some examples:

Example 1: Converting array of Strings to String

Example 2: Converting array of Int to String

Example 3: Calculate sum of array of Int

--

--

Maher Bhavsar

Maher Bhavsar

Hi, I am enthusiast in Big Data, Hadoop and Spark