YARN (Yet Another Resource Negotiator) is a core component of Hadoop that manages and allocates resources, such as CPU and memory, to applications. It helps in running a variety of data processing applications on large clusters efficiently.
The process for allocating resources in YARN can be explained through the following steps:
1. **Submitting an application**: When a user submits an application to YARN, they need to include an Application Master (AM). The Application Master communicates with the YARN ResourceManager to request and release resources (containers) for the application.
2. **ResourceManager**: The YARN ResourceManager is responsible for managing resources and making global decisions about resource allocation across all applications. It does not have detailed information about the status or requirements of individual applications. Instead, it relies on NodeManagers and Application Masters to coordinate and report back the application status.
3. **Placement Algorithm**: ResourceManager uses a Placement Algorithm, such as Capacity Scheduler or Fair Scheduler, to determine which resources should be allocated to eligible applications. The goal of these algorithms is to achieve better resource usage, making sure applications get fair access to resources and respect cluster-wide settings (quotas, resource limits, etc.).
4. **Application Master**: The Application Master of an application sends a Resource Request to the ResourceManager. The Resource Request contains details about the required resources, such as the number of containers, resource capacities (CPU, memory), and locality constraints (node, rack).
5. **NodeManager**: NodeManagers run on each worker node in the cluster and monitor resources on each node. They are responsible for managing and monitoring the status of containers that run on their respective nodes. They periodically report the status and resource usage of their assigned containers to the ResourceManager.
6. **Container Allocation**: When the ResourceManager decides to allocate resources to an application, it grants a container with the required resources. The Application Master receives a Container Allocation Grant with the details about the container, such as NodeManager address and available resources.
7. **Launching containers**: The Application Master communicates with the corresponding NodeManager to start the allocated container. The NodeManager creates an isolated environment (cgroup or JVM) and runs the requested task or process in the container.
Here’s an example to illustrate resource allocation in YARN:
Let’s assume we have three applications, App1, App2, and App3, and ResourceManager has a total of 30 cores and 100 GB of memory available for allocation.
App1 requests 10 containers, each with 2 cores and 3 GB of memory for a total of 20 cores and 30 GB of memory.
App2 requests 5 containers, each with 3 cores and 5 GB of memory for a total of 15 cores and 25 GB of memory.
App3 requests 8 containers, each with 2 cores and 4 GB of memory for a total of 16 cores and 32 GB of memory.
Assuming ResourceManager is using the Fair Scheduler, it calculates the resource shares for each application based on memory and cores. The share for each application is:
- App1: $\frac{20 \ cores + 30 \ GB}{30 \ cores + 100 \ GB}$ = $\frac{50}{130}$
- App2: $\frac{15 \ cores + 25 \ GB}{30 \ cores + 100 \ GB}$ = $\frac{40}{130}$
- App3: $\frac{16 \ cores + 32 \ GB}{30 \ cores + 100 \ GB}$ = $\frac{48}{130}$
The ResourceManager will distribute resources based on these shares, ensuring fairness across the applications.
In summary, YARN allocates resources for applications in a distributed and flexible manner. The ResourceManager, with the help of NodeManagers and Application Masters, uses allocation algorithms like Capacity Scheduler or Fair Scheduler to distribute resources efficiently and fairly.