Prabhakaran VijayanaguluinGeek CultureSubmitting spark job in Azure HDInsight through Apache LivyApache Livy is designed in a way to easily interact with any remote cluster running spark, synchronously/asynchronously, through a REST…Jun 4, 20211Jun 4, 20211
Prabhakaran VijayanaguluinTowards Data ScienceSpark 3.0 SQL Feature Update| ANSI SQL Compliance, Store Assignment policy, Upgraded query…Spark has added a lot of notable features with Spark SQL. Some will have a huge impact on checks like data quality and data validations…Sep 8, 2020Sep 8, 2020
Prabhakaran VijayanaguluSpark 3.0 Feature — Dynamic Partition Pruning (DPP) to avoid scanning irrelevant DataSpark 3.0 has introduced multiple optimization features. Dynamic Partition Pruning (DPP) is one among them, which is an optimization on…Jul 28, 20202Jul 28, 20202