Hadoop 2.0, which is commonly referred to as YARN (Yet Another Resource Negotiator), introduced many significant enhancements to the platform. One of them is the improved checkpointing mechanism in HDFS (Hadoop Distributed File System).
Checkpointing is the process of periodically saving the filesystem metadata to ensure faster and more efficient recovery in case of failures. The metadata in HDFS is managed by the NameNode, which is a master node that maintains the file system image and the log of changes (edit log) to that image.
In Hadoop 2.0, the checkpointing mechanism was enhanced with the introduction of Standby NameNodes. This enhancement makes the Hadoop platform more resilient to NameNode failures.
Here’s an overview of how checkpointing works in Hadoop 2.0:
1. Multiple copies of the file system metadata are maintained as FSImage and EditLogs.
The file system state is represented using two types of files: the FSImage and the EditLog. The FSImage is a snapshot of the Hadoop filesystem metadata, and the EditLog is a sequence of transaction logs that record the changes made to the file system.
$$\text{FSImage} = \text{Snapshot of HDFS metadata} \\
\text{EditLog} = \text{Transaction log of changes to HDFS metadata}$$
2. Checkpointing is performed by Secondary NameNode or CheckpointNode.
In Hadoop 2.0, the checkpointing process can be performed by a dedicated node, called the Secondary NameNode. The Secondary NameNode downloads the FSImage and EditLog from the primary NameNode, then merges them and uploads the new FSImage back to the NameNode.
Also, Hadoop 2.0 introduced another option to perform checkpointing using a CheckpointNode. The CheckpointNode provides the same functionality as the Secondary NameNode but allows for more flexibility and manual control over when checkpoints should be taken.
3. Standby NameNode helps in faster and smoother recovery.
Hadoop 2.0 introduced Standby NameNodes which maintain a synchronized copy of the active NameNode’s metadata in memory. This allows Standby NameNodes to quickly take over in the case of an active NameNode failure, significantly reducing the downtime of the HDFS cluster. Standby NameNodes use the JournalNode service to keep their metadata up-to-date with the active NameNode.
Here is an illustration of how the checkpointing process works in Hadoop 2.0:
$$\begin{array}{c}
\textbf{Active NameNode} \\
\downarrow \\
\text{FSImage and EditLog} \\
\downarrow \\
\underline{\textbf{Secondary NameNode / CheckpointNode}} \\
\uparrow \\
\text{Updated FSImage} \\
\uparrow \\
\textbf{Active NameNode}
\end{array}$$
This improved checkpointing mechanism in Hadoop 2.0 provides better fault tolerance and faster recovery times for the HDFS cluster, ensuring more reliable and efficient big data processing.