Geek Culture
Published in

Geek Culture

Batch processing in scale using Kubernetes jobs

This article talks about an approach that can help customers scale large file processing workloads in AWS EKS using Kubernetes jobs and AWS step functions.

Introduction

This approach uses AWS step functions to orchestrate the end-to-end flow, which involves:

  • Reading of the input file from AWS S3
  • Splitting the large input file into smaller files, processing it, saving the data into a database
  • Writing the output file to AWS S3

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store