Sergey KotlovSpark on AWS using Spot Instances: ensuring capacity and optimizing costsIn this post, I will share our experience of using AWS Spot Instances for Spark applications. We will talk about the problems you may…Jul 8, 2022Jul 8, 2022
Sergey KotlovinTowards Data ScienceMonitoring of Spark ApplicationsUsing custom metrics to detect problemsMay 17, 20223May 17, 20223
Sergey KotlovUnit Testing of Spark ApplicationsIn this post, we’ll look at one of the ways to unit test Spark applications and prepare test datasets. The motivation for using the…Mar 29, 2022Mar 29, 2022
Sergey KotlovLogging for Spark on KubernetesIn 2021 at Joom, we migrated our Spark setup from EMR to Kubernetes/EKS. With Spark on EMR, we had support for logging out of the box. But…Feb 23, 2022Feb 23, 2022