Hadoop is an open-source framework designed to store and process large volumes of data across clusters of computers. It is built on top of the Hadoop Distributed File System (HDFS) and the MapReduce programming model, which allows for parallel processing of data. Hadoop is intended to scale up from a single server to thousands of machines, each offering local computation and storage.
The Hadoop architecture consists of several components:
1. Hadoop Distributed File System (HDFS): HDFS is a distributed and scalable file system that stores large files across multiple machines and provides high-throughput access to data. It consists of two main components: NameNode and DataNodes.
- NameNode: It is a master node that manages the file system metadata, such as namespace organization, file-to-block mappings, and access information. It is responsible for maintaining the health of the cluster by keeping track of file splits, replication factor, and block locations.
- DataNodes: These are slave nodes that store the actual data blocks on local disks. They communicate with the NameNode to report block locations, execute storage-related operations, and manage data replication.
2. MapReduce: MapReduce is a programming model that simplifies distributed data processing. It consists of two main functions, Map and Reduce.
- Map: This function takes input data, processes it, and generates intermediate key-value pairs. The input data is split into chunks, which are then processed by different Map tasks running concurrently on separate nodes.
- Reduce: This function takes the intermediate key-value pairs, groups them by key, and processes them further to produce the final output. Reduce tasks are executed after all Map tasks have finished.
3. YARN (Yet Another Resource Negotiator): YARN is the cluster resource management layer of Hadoop, responsible for managing and allocating resources like CPU, memory, and disk space for applications running on the platform. It consists of two main components:
- ResourceManager: It is the main scheduler responsible for managing the cluster’s resources and assigning tasks to the nodes based on their resource needs and availability. The ResourceManager keeps track of node health and capacity, and it makes scheduling decisions to optimize resource utilization and maintain overall cluster performance.
- NodeManager: These are the per-node components running on each machine in the cluster, responsible for managing the application containers and monitoring their resource usage and health. They communicate with the ResourceManager to report node status and receive task assignments.
Here is a diagram representing the Hadoop architecture:
Hadoop delivers scalable and fault-tolerant storage and processing capabilities by distributing the data and computation across multiple nodes in a cluster, leveraging data locality to optimize performance, and replicating data blocks to ensure fault tolerance. This architecture allows Hadoop to handle large-scale data processing tasks efficiently and effectively.