YARN (Yet Another Resource Negotiator) is a critical component of Hadoop, introduced in Hadoop version 2.0 as part of the Hadoop 2.x series to overcome limitations of the previous architecture. YARN serves as a cluster resource management system, responsible for managing resources and scheduling tasks on the Hadoop cluster. It significantly improved the scalability, flexibility, and resource utilization of the Hadoop framework compared to its predecessor, Hadoop 1.x, which relied solely on the MapReduce programming model and the Hadoop JobTracker for managing resources.
YARN was primarily introduced for the following reasons:
1. Scalability: Hadoop 1.x’s initial design faced issues when scaling beyond 3,000 to 4,000 nodes and more than 40,000 tasks. YARN, on the other hand, improved Hadoop’s resource manager allowing support for larger clusters and higher workloads.
2. Resource Management: The JobTracker in Hadoop 1.x had multiple responsibilities, including job and task scheduling, and resource management. This led to a bottleneck in larger clusters. YARN divided these functions, making the ResourceManager (RM) responsible only for allocating resources, while the ApplicationMaster (AM) is in charge of job-specific coordination and task monitoring.
3. Multi-tenancy: Hadoop 1.x was limited to MapReduce jobs only. With YARN, Hadoop became a more general-purpose platform, supporting various data processing models, including MapReduce, Spark, Flink, and Tez. This allows users to run multiple applications simultaneously on the same Hadoop cluster.
4. Better Resource Utilization: In Hadoop 1.x, the JobTracker allocated fixed-size slot-based resources for Map and Reduce tasks. This was inefficient when tasks did not fully utilize the allocated resources. YARN, on the other hand, allows dynamic allocation of resources based on the actual demand, which leads to better resource utilization.
To summarize, YARN is an important component of Hadoop responsible for resource management and scheduling tasks on the cluster, overcoming limitations related to scalability, resource management bottlenecks, multi-tenancy, and efficient resource utilization that existed in the Hadoop 1.x architecture.