Jul 23, 2017 · 1 min read
I have used foreachPartitionAsync, batches get processed in milli seconds but I can see a long list in Active Jobs and they are processing a bit slower than when I was using foreach. Am I doing something wrong? Spark Scheduler Mode is FAIR.