EXPEDIA GROUP TECHNOLOGY — SOFTWARE
Part 6: Summary of Apache Spark Cost Tuning Strategy & FAQ
The step by step overview of the cost tuning strategy
If you like this blog, please check out my companion blogs about how to use persist and coalesce effectively to lower job cost and improve performance.
If you haven’t read the entire series of these Apache Spark cost tuning articles, then the changes recommended in this summary may not make sense. To understand these steps, I encourage you to read Part 1 which provides the philosophy behind the strategy, and Part 2 which shows you how to determine the estimated costs for your Spark job. The full series is linked below, after this summary of the steps you should take:
- Switch executor core count to the ideal core count for your node as described in Part 3.
- If executor core count changed then adjust executor count using the method described in Part 4.
- Change executor memory to the efficient memory size for your node as described in Part 3.
- When executor memory issues happen while running with the new config, add tweaks that resolve memory issues as described in Part 5.
- If job is running with 100% CPU utilization and 100% memory utilization, consider running the job on a node with more memory per node CPU as described in Part 4 to improve run time.
- If the run time slows down after tuning and you want to sacrifice some cost savings for run time improvement, follow the method described in Part 4 to try improving run time.
Q: What executor config do you recommend for a cluster with 32 cores and 256GB?
A: Because 31 is a prime number, I actually recommend leaving 2 cores for YARN and system processing. That leaves 30 cores for available processing which means a 5 core executor with 34GB of memory will work for this node as well.
Q: What executor config do you recommend for clusters with nodes that have 8 or fewer cores?
A: I only recommend using 8 core (or less) nodes if your Spark jobs only run on a single node. If your jobs span two 8 core nodes (or four 4 core nodes) then your job would be better served running on a 16 core node. There are many reasons for this.
- The only config that utilizes all available CPUs on an 8 core node would be a 7 core executor. The consensus of the Spark community is that 5 core executors are the most performant. I’ve confirmed this with my own testing as well. So using 7 core executors will slowed down performance.
- Two 8 core nodes will only have 14 CPUs available for use as opposed to have 15 CPUs available on a single 16 core node.
- Two 8 core nodes will cost the same as one 16 core node (in the same instance family). So this means that when you use 16 core nodes you get better performance with your executors and one additional CPU to consume for the same price as two 8 core nodes.
I will be updating this FAQ in the future when needed.
Series contents
- Part 1: Cloud Spending Efficiency Guide for Apache Spark on EC2 Instances
- Part 2: Real World Cost Tuning Examples
- Part 3: Cost Efficient Executor Configuration for Apache Spark
- Part 4: How to Migrate Existing Apache Spark Jobs to Cost Efficient Executor Configurations
- Part 5: How to Resolve Common Errors When Switching to Cost Efficient Apache Spark Executor Configurations
- Part 6: Summary of Apache Spark Cost Tuning Strategy & FAQ
