Unlocking the Power of Spark on Kubernetes with Apache YuniKorn
Author: Krishna Birla, Amogh Desai
Hadoop Meetup 2023 (Bangalore)
Introduction
On October 12, 2023, a significant event took place at the LinkedIn office in Bangalore, Karnataka. The Hadoop MeetUp featured a variety of engaging talks and discussions on cutting-edge technologies. Among them, one talk that stole the spotlight was “Unlocking the Power of Spark on Kubernetes with Apache YuniKorn,” presented by Krishna Birla and Amogh Desai from Cloudera. In this blog, we’ll delve into the key takeaways from their enlightening presentation. Let’s dive in!
The Challenges of Running Spark on Kubernetes
The speakers began by highlighting the challenges of running Apache Spark on Kubernetes with its default scheduler (an inherently services-based scheduler). Kubernetes’ default scheduler is simply not optimized for batch workloads in its crude form. To make Spark work seamlessly on Kubernetes, several challenges have to be addressed. Some of these challenges already have solutions in Kubernetes, such as job orchestration, dependency management, autoscaling and node placement, security, and networking. However, there are key notable gaps: lack of advanced batch scheduling and multi-tenancy.
Why Kubernetes Resource Scheduling Falls Short
The speakers explained why Kubernetes resource scheduling was insufficient for Spark workloads. Some of the shortcomings they highlighted included:
- No workload queuing: Kubernetes rejects workloads outright during a resource quota crunch, requiring users to implement complex retry logic.
- All workloads share the same resource pool/queue, which is the entire cluster.
- Lack of application sorting.
- No concept of applications, leading to inadequate application-aware scheduling.
- Cluster-wide preemption with limited user control.
- No concept of Gang Scheduling.
Introducing Apache YuniKorn
To address the above issues in Kubernetes, they introduced Apache YuniKorn, a Universal Resource Scheduler with advanced batch scheduling capabilities, making it the silver bullet of Kubernetes Resource Scheduling and a powerful solution for enabling Spark workloads on Kubernetes. They highlighted some key features of YuniKorn, including workload queuing, gang scheduling, application sorting, advanced preemption, user and group quota enforcement, application history, and an interactive web UI.
High Level Architecture of YuniKorn
The speakers also discussed YuniKorn’s high-level architecture, consisting of four key components:
- YuniKorn Scheduler Core: The core component responsible for placement and scheduling decisions.
- YuniKorn Scheduler Interface: A communication protocol between the scheduler core and shim layers.
- YuniKorn Shim: Translates resource requests from the host system (Kubernetes) and manages pod bindings.
- YuniKorn WebUI: An interactive web interface that provides insights into managed nodes, resources, applications, and queues.
Separate Core and Shim layers make YuniKorn a true Universal Resource Scheduler, while keeping it lightweight, fast and robust.
YuniKorn offers two deployment modes:
- Standard Mode: This mode replaces the default Kubernetes scheduler, providing a custom scheduler with full scheduling and binding logic.
- Plugin Mode: This mode works in conjunction with Kubernetes’ Scheduling Framework APIs, either replacing or augmenting limited functionality.
Gang Scheduling: A Game-Changer
The presenters emphasised the importance of Gang Scheduling in YuniKorn. This innovative feature allows Spark applications to define task groups for driver and executors. Gang scheduling ensures that these task groups are scheduled in an “all or none” fashion, preventing resource segmentation and deadlocks. Placeholder pods are allocated and scheduled first to guarantee the availability of resources, making autoscaling more efficient in cloud environments.
A task group is a “gang” of tasks in an app, these tasks have the same resource profile and the same placement constraints. They are considered as homogeneous requests that can be treated as the same kind in the scheduler. Once the job is submitted to the scheduler, the job won’t be scheduled immediately. Instead, the scheduler will ensure it gets its minimal resources before actually starting the driver/executors.
YuniKorn also offers the option to use “Soft” Gang Scheduling, which falls back to normal scheduling in case the app cannot be gang scheduled.
A Powerful Preemption Demo
During the presentation, the audience was treated to a demo showcasing the power of preemption in YuniKorn. The hierarchical queue structure, and ability to define guaranteed resources and priorities on queues demonstrated how YuniKorn efficiently manages cluster resources, ensuring optimal resource utilisation.
You can read more on YuniKorn’s advanced preemption here.
Companies Embracing Apache YuniKorn
The speakers highlighted some prominent companies that have successfully integrated Apache YuniKorn into their Kubernetes clusters:
1. Cloudera
2. Apple
3. Alibaba
4. Visa
5. Lyft
6. Zillow
Of particular interest was Amazon AWS’s recommendation to use Apache YuniKorn with Gang Scheduling for running Elastic MapReduce (EMR) on Elastic Kubernetes Service (EKS) for Spark workloads, emphasising the value of the Gang Scheduling feature. Get the full AWS guide here.
The Broader Hadoop MeetUp
In addition to the insightful talk by Krishna Birla and Amogh Desai, the Hadoop MeetUp also featured three other talks:
1. “Data Ingestion @ LinkedIn” by Bhupendra Kumar Jain.
2. “Operating Kafka At Uber Scale” by Abhijeet Kumar & Nikin Raagav.
3. “Grid Compute @ LinkedIn” by Krishan Goyal & Aditya Sharma.
These talks provided valuable insights into various aspects of data ingestion at scale, large messaging buses, and distributed computing.
Post-Talk Networking Session
Following the talks, the MeetUp concluded with a one-hour networking session where speakers and in-person attendees engaged in discussions about various topics, shared the issues faced by various stakeholders and discussed possible solutions. The talk on “Unlocking the Power of Spark on Kubernetes with Apache YuniKorn” received significant attention and garnered enthusiastic feedback. Attendees were particularly intrigued by YuniKorn’s capabilities, with a special focus on the “Gang Scheduling” feature, which led to a multitude of questions and discussions. The presenters fielded significant inquiries from representatives of various organisations regarding the implementation of “hard” Gang Scheduling in real-world production environments.
The discussions further extended to encompass highly productive dialogues on the intricacies of scaling resource scheduling for large Kubernetes clusters that have 5,000+ nodes. Additionally, there were enlightening exchanges on the subject of federating resource scheduling, offering valuable insights into the orchestration of resources across multiple Kubernetes clusters. These interactions underscored the profound interest and engagement of the audience with the transformative capabilities of Apache YuniKorn in diverse enterprise settings.
Conclusion
The Hadoop MeetUp provided a platform for tech enthusiasts to explore the latest developments in the field. Krishna Birla and Amogh Desai’s talk on “Unlocking the Power of Spark on Kubernetes with Apache YuniKorn” left a profound impact. Their discussion of the challenges in Kubernetes, possible solutions, and innovative features of YuniKorn shed light on the future of Spark workloads on Kubernetes.
In addition to this standout talk, other speakers shared valuable insights on topics like data ingestion, operating Kafka at scale, and grid computing. It’s worth noting that the Hadoop MeetUp attracted not only an in-person audience but also hundreds of attendees online. This broader reach ensured that the knowledge and insights shared during the event reached a global audience, fostering greater collaboration and learning in the tech community.
Apache YuniKorn is undoubtedly a game-changing solution for optimizing Spark workloads on Kubernetes. The Hadoop MeetUp was a great success, fostering valuable discussions and connections among tech professionals, both in person and online. The future of technology is brighter and more collaborative, thanks to events like these.
About the Speakers
Krishna Birla
Krishna Birla is a Senior Software Engineer at Cloudera, working on Resource Management and Multitenancy in Kubernetes for the Compute Platform team. He completed his bachelors degree in Computer Science from Manipal Institute of Technology in 2020 and has almost 4 years of industry experience.
Amogh Desai
Amogh Desai is a Software Engineer at Cloudera, working on Data Engineering for the Experience Resource Management team. He is an Apache Airflow Committer, and has almost 2 years of industry experience.
