How is data stored in Hadoop?

In Hadoop, data is stored using the Hadoop Distributed File System (HDFS), which is designed to handle large volumes of data and provide high-throughput access to this information. Data is stored across a cluster of machines in a distributed manner, ensuring fault tolerance and parallel processing capabilities. Let’s dive into the details of HDFS and how data is stored in Hadoop.

HDFS Principles:

1. Blocks: Files in HDFS are split into smaller chunks called blocks, by default, set to a size of 128 MB or 64 MB, but this block size can be customized. These blocks are then stored across the nodes in the Hadoop cluster, enabling data redundancy and parallel processing. The blocks are stored as separate files on the nodes’ local file systems.

2. Replication: To provide fault tolerance and high availability, each data block is replicated multiple times (by default, 3 times). These replicas are stored on different nodes of the Hadoop cluster. The replication factor can be adjusted depending on the needs of the system.

3. NameNode: The Hadoop cluster has a master node, called the NameNode, which is responsible for managing the file system metadata. The NameNode keeps track of the file-to-block mapping, block locations, and other metadata information. Users and applications interact with the NameNode to perform file system operations, such as opening files, creating directories, and renaming files.

4. DataNode: The actual data blocks are stored on worker nodes, called DataNodes. The DataNode is responsible for serving read and write requests from clients, as well as performing block creation, deletion, and replication tasks instructed by the NameNode.

When a file is written into HDFS, the file is split into blocks, and these blocks are distributed across multiple DataNodes based on the replication factor. The following diagram illustrates how a file is stored in Hadoop:

     +---------+                +----------+ 
     | NameNode|                | DataNode | 
     +----+----+                +----+-----+ 
          |                          |
          |                          | 
     +----v----+                +----v-----+ 
     |  File   |                |  Block_1 | 
     |Metadata |<-----------| (Replica)| 
     +---------+                +----------+ 
                                   | 
                            +------v------+
                            |   Block_2   |
                            |   (Replica) |
                            +-------------+
                                   |
                            +------v------+ 
                            |   Block_3   | 
                            |   (Replica) | 
                            +-------------+ 

In summary, data in Hadoop is stored in HDFS, which divides files into blocks and distributes them across the nodes in a cluster. The blocks are then replicated according to the replication factor, providing fault tolerance and high availability. The NameNode manages file system metadata, while the DataNodes store and manage the actual data blocks.