Hadoop Distributed File System (HDFS) is the storage system used by Hadoop to store large-scale data across multiple nodes in a distributed manner. It is designed to be fault-tolerant, scalable, and reliable. The architecture of HDFS comprises mainly two components: NameNode and DataNode. Let’s see these components and their roles in detail.
1. NameNode (Master Node):
NameNode is the master node in the HDFS architecture that manages the file system metadata. It is responsible for managing the namespace, maintaining the file system tree, and tracking the files’ metadata (such as file location, block location, replication factor, etc.). NameNode stores all this metadata in its memory (RAM) to enable quick access to the information. A disk-based copy of the metadata is available in a file known as the FsImage.
2. DataNode (Worker Nodes):
DataNodes are the worker nodes that actually store the data blocks as per the file system. These blocks are nothing but parts (chunks) of the file, divided so that the data can be stored in multiple DataNodes. The default Hadoop block size is 128 MB, but it can be configured as needed. DataNodes constantly communicate with the NameNode to update the metadata about their blocks and also to receive instructions (to replicate, delete, or add blocks).
In addition to NameNode and DataNodes, there are two more auxiliary components that support HDFS:
3. Secondary NameNode:
The Secondary NameNode is not a backup or high-availability NameNode. Instead, it is a helper to the NameNode. It is responsible for creating periodic check-pointing by merging the FsImage and the recent metadata changes from the EditLogs. This helps to minimize the downtime in case of NameNode’s failure, as the updated FsImage can be loaded to the NameNode. In newer Hadoop versions, the role of Secondary NameNode is replaced by CheckpointNode and BackupNode.
4. High Availability (HA) Components:
High Availability in HDFS is achieved by deploying multiple NameNodes in an Active/Passive configuration. In an HA cluster, there’s an Active NameNode and one or more Standby NameNodes. The Active NameNode performs all namespace tasks, while the Standby NameNodes receive the namespace updates continuously. In case the Active NameNode fails, one of the Standby NameNodes takes over its role seamlessly.
Now let’s visualize the overall architecture in the following diagram:
In summary, the HDFS architecture is designed to provide fault-tolerance, high availability, and scalability across multiple nodes in Hadoop clusters to store and process large-scale data sets. The core components of HDFS architecture are NameNode and DataNode, which play a crucial role in managing metadata and data blocks, respectively. Secondary NameNode and High Availability components are provided for reliability and redundancy in the system.