YARN (Yet Another Resource Negotiator) is a Hadoop component responsible for managing resources and scheduling applications running on a Hadoop cluster. YARN supports several types of schedulers, which are responsible for scheduling resources and determining how jobs and tasks are executed. Those schedulers include:
1. **FIFO Scheduler (First-In-First-Out)**
FIFO Scheduler is the simplest scheduler in YARN, where applications are executed as per their arrival sequence without considering any resource requirements. This scheduler is easy to set up and works well when there are only a few applications to execute and resources are evenly distributed. However, it lacks several important features such as resource management, data-locality optimizations, and support for multi-tenancy.
2. **Capacity Scheduler**
Capacity Scheduler is designed to efficiently allocate resources in a multi-tenant environment. It allows organizations to share a large cluster while guaranteeing each tenant a specific capacity (percentage of the total resources). The main features of the Capacity Scheduler include:
Hierarchical Queues: Users can submit jobs to various queues that have dedicated capacities. This simplifies the organization of the resources to different users and groups.
Fair Sharing and Preemption: This scheduler supports fair sharing within a queue and preemption among queues. It allows for better utilization of resources and reduces the waiting time for high-priority applications.
Multi-resource Scheduling: Capacity Scheduler supports scheduling resources such as CPU and memory, providing better resource utilization and flexibility.
Scalability: The scheduler is designed to work well in large clusters, offering better performance and scalability compared to the FIFO Scheduler.
3. **Fair Scheduler**
Similar to the Capacity Scheduler, the Fair Scheduler is designed for multi-tenant environments. However, the main focus of Fair Scheduler is to ensure that all applications receive a fair share of resources over time. The main features of the Fair Scheduler include:
Fairness: The scheduler dynamically adjusts the resources allocated to each job, ensuring they receive an equal share of cluster resources over time.
Weight-based Priorities: This scheduler allows assigning weights to users and applications, which are used in determining the fair share of resources for each application or user.
Pool Management: Fair Scheduler allows creating pools that contain a set of applications belonging to a single user or group. Each pool receives a share of cluster resources, and applications within a pool share the pool’s resources.
Preemption: Just like the Capacity Scheduler, the Fair Scheduler also supports preemption. When resources are under-utilized, the scheduler can preempt resources from lower-priority applications and allocate them to higher-priority applications.
In conclusion, each scheduler available in YARN provides specific features and capabilities to manage and allocate resources in Hadoop clusters. Your choice of scheduler will depend on your use case, requirements, and the level of granularity needed in managing resources at the cluster, queue, and application level.