YARN (Yet Another Resource Negotiator) is a resource management framework for Hadoop that was introduced in Hadoop version 2.x. YARN enables the management and scheduling of resources in a distributed and scalable big data ecosystem.
There are four key components of YARN:
1. **Resource Manager**: The Resource Manager is the central authority that manages resources and schedules applications in a Hadoop cluster. It is composed of two primary components:
- Scheduler: The Scheduler allocates resources to the various running applications in the cluster based on the specified resource requirements.
- Application Manager: The Application Manager helps bootstrap and restart Application Masters in case of hardware or software failures.
2. **Node Manager**: The Node Manager is a per-node agent that runs on each worker node. It helps manage resources and execute tasks on the individual machines in the cluster. The Node Manager is responsible for:
- Monitoring resources, such as CPU, memory, disk, and network usage of the applications running on the node.
- Launching, monitoring, and cleaning up containers as directed by the Resource Manager.
- Communicating between the Resource Manager and the containers.
3. **Application Master**: The Application Master runs one instance per-application and coordinates distributed execution across all allocated containers on the Node Managers. Each application has a dedicated Application Master that:
- Negotiates resources for the application with the Resource Manager.
- Works with the Node Manager to execute tasks within containers.
- Monitors application progress and handles failures.
4. **Container**: A container is a logical unit representing the collection of allocated resources like CPU, memory, and disk space across multiple nodes. Containers are responsible for running application-specific tasks, such as map and reduce tasks in MapReduce, or executor tasks in Spark.
The YARN architecture can be illustrated as follows:
In summary, YARN provides an efficient and scalable way to manage resources in a Hadoop cluster, allowing multiple applications and workloads to coexist and share resources efficiently. Its key components are the Resource Manager, Node Managers, Application Masters, and Containers that work together to enable scheduling and execution of big data applications in a distributed environment.