Adding and removing nodes in a Hadoop cluster involves multiple steps. Let’s discuss each one separately.
1. **Adding Nodes**
To add a new node to a Hadoop cluster, follow these steps:
Step 1:* Install the necessary software (‘Hadoop‘ and ‘Java‘) on the new DataNode.
Step 2:* Configure the ‘core-site.xml‘, ‘hdfs-site.xml‘, and ‘yarn-site.xml‘ files on the new DataNode. Make sure the new DataNode points to the same NameNode address as the existing DataNodes.
Step 3:* Add the new DataNode’s IP address or FQDN (Fully Qualified Domain Name) to the ‘slaves‘ file located in the Hadoop configuration directory on the NameNode and the ResourceManager node.
Step 4:* Start the DataNode daemon and NodeManager daemon on the new DataNode. Use the following commands:
hadoop-daemon.sh start datanode
yarn-daemon.sh start nodemanager*Step 5:* Monitor the Hadoop cluster using the NameNode web UI or ResourceManager web UI to ensure the new DataNode is functioning properly.
2. **Removing Nodes**
To remove a node from a Hadoop cluster, follow these steps:
*Step 1:* Stop the DataNode daemon and NodeManager daemon on the node you want to remove. Use the following commands:
hadoop-daemon.sh stop datanode
yarn-daemon.sh stop nodemanager*Step 2:* Remove the decommissioning node’s IP address or FQDN (Fully Qualified Domain Name) from the ‘slaves‘ file on the NameNode and ResourceManager node.
*Step 3:* Add the decommissioning DataNode’s IP address or FQDN to the ‘dfs.hosts.exclude‘ file on the NameNode.
*Step 4:* Refresh the list of DataNodes by executing the following commands:
“‘ hdfs dfsadmin -refreshNodes yarn rmadmin -refreshNodes “‘
*Step 5:* Monitor the Hadoop cluster using the NameNode web UI or ResourceManager web UI to ensure the decommissioning DataNode is not listed and its data is replicated to other DataNodes.
*Step 6:* Once the decommissioning is complete and the DataNode is no longer part of the cluster, you can stop the services on the removed node and delete the data directories.
In summary, adding a node to a Hadoop cluster involves installing the necessary software, configuring the Hadoop configuration files, and adding the new DataNode’s IP address or FQDN to the existing cluster. Removing a node from a Hadoop cluster involves stopping the DataNode and NodeManager daemon, updating the exclusion list, refreshing the list of DataNodes, and deleting the data directories.