YARN (Yet Another Resource Negotiator) is the resource management layer of Hadoop that intelligently manages resources and job scheduling across the cluster. Job scheduling in YARN is accomplished through a combination of components: the ResourceManager, the NodeManagers, and the ApplicationMaster.
Let’s understand the roles of these components first:
1. **ResourceManager**: The ResourceManager is the central authority that arbitrates resources among all the applications in the cluster. It manages the global assignment of compute resources for all submitted applications, including starting ApplicationMasters for each application.
2. **NodeManagers**: NodeManagers run on the Hadoop cluster’s compute nodes, managing the resources and containers on each node. Containers represent a collection of resources, such as memory, CPU, and storage, that can be allocated to applications.
3. **ApplicationMaster**: The ApplicationMaster (AM) is per-application and is in charge of negotiating resources (containers) from the ResourceManager and executing tasks within those containers. It manages the application’s lifecycle, including starting, scaling, and stopping tasks.
Now let’s discuss the job scheduling process in YARN:
1. **Submit Application**: The client submits an application to the ResourceManager, which includes the ApplicationMaster’s specification and resources required for the application’s tasks.
2. **Start ApplicationMaster**: The ResourceManager selects a suitable NodeManager and allocates a container for the ApplicationMaster to start.
3. **Resource Request**: The ApplicationMaster communicates with the ResourceManager to request containers for its tasks, specifying the resources required (e.g., memory, CPU, storage).
4. **Allocate Containers**: The ResourceManager analyzes the available resources in the cluster (tracked through NodeManagers) and allocates containers for the ApplicationMaster based on resource requirements and the scheduling policy.
5. **Launch Task**: The ApplicationMaster contacts the NodeManager running the allocated container and requests it to launch the task within the container.
6. **Monitor Progress**: The ApplicationMaster monitors the progress of tasks and can ask for more or fewer resources as needed. In the event of task failure, the AM requests additional resources from the ResourceManager to restart the task.
7. **Complete Application**: Once all tasks are completed, the ApplicationMaster informs the ResourceManager that the application has finished. The ResourceManager releases resources used by the ApplicationMaster, and the AM shuts down.
YARN supports several application-level schedulers and schedulers that decide how resources are to be allocated to applications:
1. **Capacity Scheduler**: The Capacity Scheduler is designed to allocate resources based on a predefined queue hierarchy with specific resource capacities. It allows sharing resources efficiently among multiple organizations, departments, or teams while maintaining security and isolation between each queue.
2. **Fair Scheduler**: The Fair Scheduler aims to allocate resources fairly among all running applications, such that each application gets, on average, an equal share of resources over time. It also supports hierarchical scheduling and priority-based preemption, allowing fast jobs to finish quickly while not starving long-running jobs.
In summary, job scheduling in YARN is comprised of ResourceManager, NodeManagers, and ApplicationMasters to manage the lifecycle of applications and distribute resources across the cluster. The Capacity Scheduler and Fair Scheduler are utilized to determine resource allocations based on predefined policies.