High availability (HA) is a crucial feature introduced in Hadoop 2.0, addressing one of the major limitations in Hadoop 1.0—the single point of failure (SPOF) in the Hadoop Distributed File System (HDFS). In Hadoop 1.0, the NameNode was responsible for managing the file system metadata, and there was only a single active NameNode in the cluster. If it failed or became unresponsive, the entire HDFS would become inaccessible, leading to potential data loss and downtime.
The HA feature in Hadoop 2.0 addresses this problem by allowing multiple NameNodes to be active simultaneously, ensuring minimal downtime and data unavailability in case of a failure. The concept of HA is essential for mission-critical applications that require continuous access to data and high reliability.
The high availability feature in Hadoop 2.0 is implemented using two NameNodes:
1. **Active NameNode**: Handles client operations, manages the file system metadata, and communicates with DataNodes.
2. **Standby NameNode**: Maintains an up-to-date copy of the file system metadata and is in read-only mode, ready to take over the role of the Active NameNode in case of failure.
Both NameNodes use a shared storage system, typically referred to as **Quorum Journal Manager (QJM)**, to store the edit logs. This ensures that the Standby NameNode remains up-to-date with the metadata changes made by the Active NameNode. Concurrently, a **ZooKeeper** service cluster is employed to manage the Active and Standby state transitions and ensure that only one NameNode is in Active mode at any given time.
The entire architecture can be visualized as follows:
Clients <--> | Active NameNode | <--> Quorum Journal Manager
| (Write) | (Shared Storage)
ZooKeeper | |
(Failure Detection) |
| Standby NameNode | <--> Quorum Journal Manager
| (Read-only) | (Shared Storage)In summary, the significance of high availability in Hadoop 2.0 lies in ensuring that the HDFS maintains continuous data access and minimizes downtime due to NameNode failures. By allowing multiple NameNodes to run simultaneously, this feature mitigates the single point of failure and provides a robust big data management system for mission-critical applications in Hadoop.