Batch processing in scale using Kubernetes jobs
This article talks about an approach that can help customers scale large file processing workloads in AWS EKS using Kubernetes jobs and AWS step functions.
This approach uses AWS step functions to orchestrate the end-to-end flow, which involves:
- Reading of the input file from AWS S3
- Splitting the large input file into smaller files, processing it, saving the data into a database
- Writing the output file to AWS S3