Hadoop 1.0 to Hadoop 2.0 saw several significant changes in data storage, addressing some of the limitations and issues present in Hadoop 1.0. Some of the major changes include the introduction of YARN, HDFS Federation, HDFS High Availability, and support for Windows.
1. Introduction of YARN:
Hadoop 1.0 used the MapReduce framework, wherein both resource management and data processing were tightly coupled. In Hadoop 2.0, YARN (Yet Another Resource Negotiator) was introduced as a new layer, which decoupled resource management and data processing functions. This change allowed Hadoop to support other processing models besides MapReduce, such as Apache Tez, Apache Flink, and Apache Spark.
YARN architecture consists of three main components:
- ResourceManager, which manages cluster resources and schedules the application.
- NodeManager, which runs on every DataNode and manages the containers, monitors resource usage, and reports the status back to the ResourceManager.
- ApplicationMaster, which manages the life cycle of each application and coordinates with ResourceManager for resource allocation.
2. HDFS Federation:
In Hadoop 1.0, there was a single Namespace and a single Block Pool managed by the NameNode. This setup became a bottleneck for large-scale Hadoop clusters. Hadoop 2.0 introduced HDFS Federation, which allows multiple namespaces and block pools to coexist, managed by separate NameNodes. The Hadoop cluster is partitioned horizontally to support multiple NameNodes. Each NameNode manages its own namespace, and the namespaces are independent of each other. This setup improves the scalability and performance of HDFS.
3. HDFS High Availability:
Hadoop 1.0 had a single point of failure in the NameNode, which could lead to downtime in case of a failure. To address this issue, Hadoop 2.0 introduced HDFS High Availability, in which there are two NameNodes: Active and Standby. Both NameNodes share metadata using a shared storage like NFS or Quorum Journal Manager (QJM). When the Active NameNode fails, the Standby NameNode takes over the workload with minimal disruption.
4. Support for Windows:
Hadoop 2.0 added support for running Hadoop on the Microsoft Windows operating system. This made it more accessible for users and enterprises that rely on the Windows environment.
In conclusion, data storage in Hadoop 2.0 became more scalable, flexible, and reliable than Hadoop 1.0 with the introduction of YARN, HDFS Federation, HDFS High Availability, and support for a broader range of platforms.