Reducing AWS EMR costs with Spot, Task Nodes, and Instance Types

Prad Pagadala, Software Engineer at 23andMe

AWS Elastic Map Reduce (EMR), a popular big data platform, powers a lot of computation at 23andMe. One particular use case is to find associations between genes and traits using Hail, which is built on top of Apache Spark. In the course of figuring out how to optimize processing for this application, we discovered ways to reduce costs significantly. EMR and Spark is a popular approach, but configuring them optimally is overwhelming. For those of you looking to accomplish this, we hope this saves you the days and dollars we had to spend!

Let’s assume that you’ve already tackled the custom tuning (cluster size, executor size, overhead, etc.) of the cluster to your application.

EMR and Spot instances:

AWS Spot instances are excess AWS compute capacity offered at up to a 90% discount. The catch: the resources can be taken away — so your system needs to be able to handle the interruption (loss) of nodes.

AWS Spot pricing works by assigning a discount relative to the on-demand price for each particular instance type. As long as instances are available, they are assigned to bidders. As soon as instances are no longer available, AWS begins interrupting current Spot holders at random. There is very little visibility into this, but the Spot Instance Advisor dashboards give you an idea of what instance types are generally available and at what discount.

Configuring EMR with Spot can provide considerable savings, but it depends on what nodes are assigned Spot instances. The primary node types in an EMR cluster are Master, Core, and Task nodes. Task Nodes are the easiest to use with Spot because they don’t store any data for the Hadoop Distributed File System, and because EMR doesn’t run any application master processes on the Task Nodes. Core nodes can also be used with Spot, but with caution. Since they usually store data, your application should be able to handle some data loss (AWS guidelines here).

Here is an example configuration for a Task Fleet for the `TestCluster` EMR cluster:

GWASTaskFleet:
  Type: AWS::EMR::InstanceFleetConfig
  Properties:
    Name: TestTaskFleet
    ClusterId: !Ref TestCluster
    InstanceFleetType: TASK
    LaunchSpecifications:
      SpotSpecification:
        TimeoutAction: TERMINATE_CLUSTER
        TimeoutDurationMinutes: 120
    InstanceTypeConfigs:
      — InstanceType: "r5.24xlarge"
        WeightedCapacity: 96
    TargetSpotCapacity: 960

Instance Fleets:

High demand for a particular instance type can cause the supply of these instances to dry up. If instances for a particular type are not available, you will not be able to get instances assigned to your cluster. A great way of reducing this risk is to diversify your instance types; this is especially straightforward with instances of the same family such as r5.

But since the demand for instances within the same family tends to be correlated, it’s important to diversify, by using different families of instances. This includes older generations of instance families, such as specifying r4 instances in addition to r5 ones. EMR has support for instance fleets of up to 5 different types. AWS will then decide which instances they will use to fulfill your requirements while maximizing capacity and minimizing cost based on how you assign weights to each instance type. Assigning weights based on the number of virtual CPUs is a metric that has been useful to us, but it depends on your executor configurations.

InstanceTypeConfigs:
  - InstanceType: "r5.24xlarge"
    WeightedCapacity: 96
  - InstanceType: "r5.12xlarge"
    WeightedCapacity: 48
  - InstanceType: "r5d.24xlarge"
    WeightedCapacity: 96
  - InstanceType: "r5d.12xlarge"
    WeightedCapacity: 48

Note: Specifying multiple different subnets allows you to utilize more AWS availability zones, helping with capacity greatly. AWS documentation on how to set this up here.

Instance Types:

Instance Type selection for EMR and Spot is quite different from EC2. For EC2, you are charged linearly with the amount of compute used; i.e., twice the compute is always twice the cost. In EMR, you are charged an EMR premium over the EC2 price per instance that doesn’t scale linearly. It’s significantly cheaper to use larger nodes because EMR pricing per instance scales sublinearly with relation to compute.

Several different instance type families can be used with EMR, but, for illustrative purposes, let’s consider the r5 family of instances:

As you can see EMR costs are flat relative to node size. This means we can use larger nodes without incurring proportional costs! This is especially useful with regards to Spot instances, because, with the Spot instance discount, EMR makes up a large proportion of the total cluster cost.

Here’s an example:

Let’s say we compare a cluster of 20 r5.12xlarge vs 10 r5.24xlarge instances, and assume a conservative Spot price discount of 60%:

EC2 only pricing:

r5.12xl = $3.024/hr * 0.4 = ~$1.21/hr
r5.24xl = $6.048/hr * 0.4 = ~$2.42/hr

20x r5.12xl = 10x r5.24xl = $24.20/hr

Let’s include the EMR premium:
20 r5.12xl = $24.20/hr + $(20*0.27)/hr = $29.60/hr
10 r5.24xl = $24.20/hr + $(10*0.27)/hr = $26.90/hr

This is a ~10% cost increase. The difference between even smaller nodes and larger ones are far more significant.

60 r5.4xls is far worse at $39.32 for the same compute capacity. Which is ~46% more expensive!

We managed to shave off around 30% on the cost of our cluster by making use of larger instance types.

Another positive effect of using larger instances is more efficiency for your application. If there is an exchange of data between executors, there is less network load as more of the traffic is within an instance. The drawback is that it exposes you to a loss of more executors on the interruption of a node, but this is greatly mitigated by having a diverse instance fleet and using Task nodes.

TL;DR

1. Use Spot instances as much as possible
2. Use Task nodes as much as possible
3. Use a diverse instance fleet for each type of node
4. Use larger instances if you can
5. Check-in with Spot advisor regularly to see what availability and discount each instance provides

About the Author

Prad Pagadala is a Software Engineer at 23andMe working on Genome-Wide Association Study (GWAS) pipelines.

23andMe is hiring! Check out our current openings!