YARN’s Capacity Scheduler is a pluggable scheduler for Hadoop Resource Manager (RM) that enables multi-tenant clusters by allocating resources based on capacity guarantees, fairness, and constraint rules. The primary goal of the Capacity Scheduler is to support large Hadoop clusters shared by multiple organizations while providing quality of service (QoS) guarantees and maximizing cluster utilization.
The Capacity Scheduler works around the concept of queues. Each queue is configured with a certain fraction of the cluster resources. The resources available in the cluster are organized into a tree-like hierarchical structure where each level represents a queue. The main characteristics of the Capacity Scheduler are:
1. Hierarchical Queues: The resource allocation rules allow users to create queues as children of other queues, thus enabling a multi-level queue hierarchy. Queues can be used to isolate different applications and user groups, providing a guaranteed share of resources for each queue.
2. Queue Capacities: Each queue has a configured capacity, expressed as a percentage of the cluster resources. The Capacity Scheduler guarantees that a queue’s active applications will get at least that much of the cluster resources when they request them.
3. Elastic Allocation: Queues can exceed their capacity if there is unused resource capacity in the cluster, thus allowing higher overall cluster utilization.
4. Preemption: When an application in a queue is waiting for resources, the Capacity Scheduler can preempt, or take back, resources allocated to other queues to meet its capacity requirements. Preemption is particularly useful when a higher-priority application requires resources.
5. Fairness: Within a queue, resources are allocated to applications with a goal to maximize fairness. Fairness is typically achieved using the concept of "fair shares," which is ensuring that applications get resources as per their respective demands, up to their maximum allowed limits.
6. Access Control Lists (ACLs): The Capacity Scheduler supports ACLs to define user and group permissions on queues. This enables resource sharing across multiple organizations while maintaining the necessary access controls.
Here is an example of a Capacity Scheduler configuration:
<name>yarn.scheduler.capacity.root.queues</name>
<value>queueA,queueB</value>
<name>yarn.scheduler.capacity.root.queueA.capacity</name>
<value>40</value>
<name>yarn.scheduler.capacity.root.queueB.capacity</name>
<value>60</value>In the configuration above, there are two main queues (queueA and queueB) under the root queue. queueA is allocated 40
In conclusion, YARN’s Capacity Scheduler allows large Hadoop clusters to be shared effectively across multiple users and organizations by providing resource guarantees, fairness, and constraint rules. This facilitates better management and utilization of resources while maintaining isolation between applications and users.