How to resolve memory issues that happen when switching to efficient executor configs — When switching to the cost efficient executor configuration, sometimes your tasks will fail due to memory errors. In this blog, I will mention three fixes you can try whenever facing these errors. Before I do that, I will cover how to lookup task failures in the Spark U/I. …

Steps to follow when converting existing jobs to cost efficient config — There are a number of things to keep in mind as you tune your Spark jobs. The following sections cover the most important ones. Which jobs should be tuned first? With many jobs to tune, you may be wondering which jobs to prioritize tuning first. Jobs that only have one or two cores per executor make…

Find the most cost efficient executor configuration for your node — CPUs per node The first step to determine an efficient executor config is to figure out how many actual CPUs (i.e. not virtual CPUs) are available on the nodes in your cluster. To do so, you need to find out what type of EC2 instance your cluster is using. For our discussion here…

I outline the procedure for working through cost tuning — Below is a screenshot highlighting some jobs at Expedia Group™ that were cost tuned using the principles in this guide. I want to stress that no code changes were involved, only the spark submit parameters were changed during the cost tuning process. …

How I saved 60% of costs in an Apache Spark job, with no increase in job time and no decrease in data processed — Until recently, most companies didn’t care how much they spent on their cloud resources. But in a covid-19 world, companies like Expedia Group™ are reducing cloud spending where reasonable. …

Best practices for detecting bad data before it spreads — In an age when HomeAway processes petabytes of data on a daily basis, data quality is critical to ensuring the right decisions are being made with our data. But how does one know if their data is good? If polluted data gets introduced into an upstream data source, then downstream…