In Hadoop, jobs are managed and scheduled using several components. Primarily, we use the Hadoop YARN (Yet Another Resource Negotiator) framework, which manages distributed resources and schedules jobs in the Hadoop ecosystem. YARN is responsible for resource allocation and job management in Hadoop. The major components of YARN are the ResourceManager, the NodeManagers, and the ApplicationMaster.
1. **ResourceManager (RM):** This component acts as the master for resource allocation. It arbitrates the resources among all submitted applications and tracks the progress of each application.
2. **NodeManager (NM):** NodeManagers reside on each DataNode in the cluster and are responsible for managing the resources on a single node, such as CPU, memory, and disk. They are in constant communication with the ResourceManager, regularly updating the ResourceManager with the status of available resources.
3. **ApplicationMaster (AM):** ApplicationMaster is responsible for managing the lifecycle of an application and negotiating resources with the ResourceManager. There is one ApplicationMaster per application running in the cluster.
Job scheduling in Hadoop involves several steps:
A. **Job Submission:** The client submits a job to the ResourceManager, providing information such as the location of input and output data, the JAR file containing the implementation, the main class to be executed, etc.
B. **Application Initialization:** Upon receiving the job, the ResourceManager assigns a unique Application ID and launches an ApplicationMaster. It allocates resources on a NodeManager for the ApplicationMaster to run. The NodeManager then launches a container to run the ApplicationMaster.
C. **Resource Request and Allocation:** The ApplicationMaster calculates the resources needed to complete the job, such as the number of map and reduce tasks and the amount of memory and CPU for each task. It requests these resources from the ResourceManager, which in turn, allocates suitable containers based on available resources.
D. **Task Execution:** Once the resources are allocated, the ApplicationMaster contacts the assigned NodeManagers and instructs them to launch the task containers. These tasks run in parallel, processing the input data and generating intermediate results. Map tasks process the input data, while reduce tasks process the intermediate results.
E. **Progress and Monitoring:** During execution, the ApplicationMaster regularly monitors the progress of the tasks and communicates with the ResourceManager to request additional resources or release resources, if necessary. The client can also query the ResourceManager or the ApplicationMaster to track the progress of the job.
F. **Job Completion:** Once all tasks complete, the ApplicationMaster informs the ResourceManager that the job is done. The ResourceManager then releases the ApplicationMaster’s resources, and the client receives a status update that the job is complete.
Scheduling in YARN can be configured with different scheduling policies using its pluggable scheduler component. Two popular schedulers in YARN are:
- **FIFO Scheduler:** Simple first-in, first-out scheduler that schedules jobs based on the order of submission. - **Capacity Scheduler:** Assigns resources based on the capacity of each queue, ensuring a fair share of resources for multiple users and groups.
These schedulers can be configured based on various factors such as Queue configuration, Priority, and Resource requests.
In conclusion, to manage and schedule jobs in Hadoop, one needs to use the Hadoop YARN framework. YARN’s ResourceManager, NodeManagers, and ApplicationMaster work together to allocate resources, execute tasks, and monitor the progress of the jobs. Additionally, various scheduling policies further enhance the scheduling mechanism in Hadoop.