The YARN (Yet Another Resource Negotiator) architecture is a significant contributor to Hadoop’s big data processing capabilities. It functions as an efficient resource manager, workload scheduler, and resource allocation system for Hadoop clusters. YARN separates the resource management and job processing components so that a wide range of applications can efficiently utilize resources while also processing large-scale data sets.
The YARN architecture comprises of four critical components:
1. ResourceManager (RM): This is the central component responsible for managing system-wide resources and allocating them to specific applications when requested. The ResourceManager oversees various activities like tracking node health and node resource usage. It comprises of two main components:
i. Scheduler: It is a pluggable component that allocates cluster resources to applications based on the configured policies like First-In-First-Out (FIFO), Capacity, and Fair Scheduler, without tracking their actual progress.
ii. ApplicationManager: It is responsible for handling new application submissions, negotiating their first container, and restarting applications that have failed due to non-fatal errors.
2. NodeManager (NM): The NodeManager exists on each worker node in the Hadoop cluster and manages the lifecycle of individual containers. Each node in the cluster has a single NodeManager that communicates with the ResourceManager, monitors resource usage (CPU, memory, disk, and network) on its node, and tracks the health of the containers executed within it.
3. ApplicationMaster (AM): For each submitted YARN application, there is an associated ApplicationMaster that coordinates their resources on the cluster. The ApplicationMaster communicates with the ResourceManager to request necessary containers/resource allocations on the worker nodes. It is also responsible for executing the application-specific tasks in those allocated containers and monitoring their progress.
4. Container: A container represents a collection of physical resources (CPU, RAM, disk space) on a single node that hosts application tasks. It isolates resources and creates an execution environment required for the application. The ApplicationMaster requests containers from the ResourceManager, and the Scheduler allocates containers based on the configured policies.
Here’s a high-level overview of the YARN architecture components:
In summary, YARN forms a critical component of the Hadoop ecosystem, laying the groundwork for efficient resource allocation, scheduling, and management. Its architecture allows for better data processing and increased scalability, making it an essential part of big data processing platforms.