Serverless Spark

Leveraging Apache Airflow for defining Data Pipelines for Spark Apps — Table Of Contents +Overview + Introduction To Apache Airflow + Serverless Spark Submission — How does Airflow fit? + Serverless Pipeline Sample DAG + Couple of Beginner Tips + A Realistic Workflow — Sales Data Analysis + Side Note + Source Code + References Overview Workflows are an indispensable element of building pipelines for…

Terraform — Overview Terraform is an open source software from Hashicorp that enables you to easily provision and manage resources of your cloud provider using a simple declarative json like configuration files to define your requirements. Terraform simplifies the provisioning aspects especially when you have to deal with multi clouds and multiple environments…

Learn how to tune logging : Executor vs Driver logs, Log Levels and Turning off Logs Overview In Part1 of this series we saw how the hooking up Log DNA with Analytics Engine service enables you to do simple troubleshooting and see the output of your print() or show() commands in…

Export Logs for each application Overview In Part1 and Part2 of this series, we saw some fundamental usage of logging in Analytics Engine Serverless Spark. For simple applications, you can look at the logDNA interface for troubleshooting or analyzing how it performed. …