Running PySpark on EKS Fargate: Part 1

It will be a series of three blogs where we will learn how to run PySpark jobs on AWS EKS Fargate.

Running PySpark on EKS Fargate

The contents will be spread over the blogs as follows:

  1. Part 1: Prerequisites and Playing with Docker Image
  2. Part 2: Manually Merging one version of Spark with that of Hadoop
  3. Part 3: Running the actual spark Job.




The publication aims at extracting, transforming and loading the best medium blogs on data engineering, big data, cloud services, automation, and dev-ops.

Recommended from Medium

Giving Our Enemies an Avoidable Homing Attack

➤鬼灭之刃 剧场版 无限列车篇 完整版本 (2020-HD) Kimetsu no Yaiba: Mugen Ressha-Hen 完整版觀看電~看电影.

Dell Poweredge 2850 Raid Controller Driver

Sharing is caring, even in coding

squirrels sharing a celery

5 Things you should avoid while doing microservices

Variables — What are they?

How we tackle Continuous Delivery at Ibuildings

New in Aiir: Simpler Programme Images

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aman Ranjan Verma

Aman Ranjan Verma

Data engineer @Flipkart, I post weekly.

More from Medium

Data Preprocess with AWS Glue

Databricks Certified Associate Developer — Apache Spark 3.x

How to resolve issues with Multiple Glue RDS Connections: 1 Trick That Works

Load data into Redshift from S3