The Hadoop 2.0 architecture includes a distributed computing model with two main components: ResourceManager and NodeManager. These components work together as part of the Hadoop YARN (Yet Another Resource Negotiator) framework to manage and allocate resources to applications running on the Hadoop cluster.
**ResourceManager (RM):** The ResourceManager is the central master component of YARN that manages resources in the Hadoop cluster, and its main responsibility is resource allocation and application lifecycle management. It has two main sub-components - Scheduler and ApplicationsManager.
1. Scheduler: The Scheduler manages resources in the cluster by allocating them to running applications, based on resource requirements, priorities, and defined policies (like fair or capacity scheduling). It tracks the resources available and makes scheduling decisions based on this information. Note that it does not monitor the applications themselves.
2. ApplicationsManager: The ApplicationsManager accepts job submissions and handles the initialization of application-specific protocols. It negotiates the first container during the application start and then launches the ApplicationMaster.
The ResourceManager receives resource requests from applications and matches them with available resources on the cluster, considering the scheduling policy and resource constraints. It also communicates with the NodeManagers to monitor the health and resource usage of the nodes.
**NodeManager (NM):** The NodeManager runs on each node (machine) in the Hadoop cluster and is responsible for managing resources and tasks on that specific node. It communicates with the ResourceManager to negotiate for and allocate resources based on application needs. It has several main responsibilities:
1. Container Management: The NodeManager launches and monitors containers, which are the basic unit of resource allocation in YARN. Containers execute various tasks of an application and have a specific amount of resources (memory, CPU, disk, etc.) assigned to them by the ResourceManager.
2. ApplicationMaster (AM) Communication: The NodeManager communicates with the ApplicationMaster to provide status updates, container requirements, and progress of the application, while also receiving commands and instructions.
3. Health Monitoring: The NodeManager continuously monitors the health of the node (memory, CPU, disk usage, etc.) and reports this information back to the ResourceManager.
4. Log Management: The NodeManager manages logs for the tasks running on the node, storing them for troubleshooting and analysis.
The ResourceManager and NodeManager work together to create a dynamic and flexible resource allocation system in Hadoop 2.0, allowing applications to efficiently utilize the resources across the cluster while maintaining fairness, isolation, and other defined policies.