YARN (Yet Another Resource Negotiator) is a critical component in the Hadoop ecosystem, responsible for managing resources and scheduling tasks in a distributed environment. Introduced in Hadoop 2.x, YARN separates the resource management and job scheduling functionalities from the data processing component, which was previously done by Hadoop MapReduce in the Hadoop 1.x version.
In YARN architecture, the two main components responsible for the efficient management of resources and applications are the ResourceManager and NodeManager:
1. ResourceManager (RM): The ResourceManager is a global authority that manages resources across the entire Hadoop cluster. It allocates and deallocates resources to different applications based on their priority, usage, and availability. The ResourceManager maintains three primary components:
- Scheduler: It allocates resources to various applications based on the predefined scheduling policy (like Fair Scheduler, Capacity Scheduler) without considering the application-specific requirements.
- ApplicationManager: It accepts job submissions, manages the application lifecycle, and ensures the application’s restart on failure.
- ResourceTrackerService: It communicates with NodeManagers to maintain an overview of available resources in the cluster.
2. NodeManager (NM): The NodeManager is a per-node agent that monitors resource usage (memory, CPU, disk, throughput) and informs the ResourceManager about the available resources and their usage. Each NodeManager registers with the ResourceManager, reports its available resources, and receives the tasks assigned for execution. The NodeManager also maintains certain components:
- ContainerExecutor: It is responsible for managing the actual tasks’ lifecycle within a container (launch, monitor, and terminate).
- ContainerLauncher: It starts the containers and assigns the required resources.
- ApplicationMaster: It is a per-application agent, responsible for coordinating between the ResourceManager and application tasks running on NodeManagers. It negotiates resources and acquires containers from the ResourceManager to execute application tasks.
In conclusion, YARN improves the scalability and flexibility of the Hadoop ecosystem by decoupling resource management and data processing, making it possible to execute a wide range of data processing applications (like MapReduce, Spark, Tez) on a Hadoop cluster.
Here’s a simple illustration of the YARN architecture: