Setup for Hadoop

This file contains instructions to install and setup Hadoop for assignment B1.

Prerequisites

OpenJDK 17
ssh
curl
gpg (optional)

Installing OpenJDK 17:

sudo apt update # Update packages
sudo apt install -y openjdk-17-jdk gpg curl ssh # Install OpenJDK 17
java -version # Check java (JRE) version
javac -version # Check javac (JDK) version

Setup Hadoop user:

sudo adduser --home /home/hadoop hadoop # Set default password as Pass@123
sudo usermod -aG sudo hadoop
su - hadoop
# Hit enter to skip entering all the information such as Full name, Room Number, etc.

SSH setup:

ssh-keygen -t ed25519 # Hit enter to save in defauly location. You can skip setting a passphrase for simplicity
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 640 ~/.ssh/authorized_keys
ssh localhost # You will be asked for conformation, type "yes"
su - hadoop

Download Hadoop (version 3.4.1 in this case):

cd ~
curl -o hadoop-3.4.1.tar.gz https://dlcdn.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz # Download Hadoop
curl -o hadoop-3.4.1.tar.gz.sha512 https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz.sha512 # Download the hash
curl -o hadoop-3.4.1.tar.gz.asc https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz.asc # Download the signature
curl -o KEYS https://downloads.apache.org/hadoop/common/KEYS # Download the keys

Verify the hash and extract the tarball:

gpg --import KEYS # Import the keys
gpg --verify hadoop-3.4.1.tar.gz.asc # Verify hash signature
sha512sum -c hadoop-3.4.1.tar.gz.sha512 # Verify hash for file

tar -xvf hadoop-3.4.1.tar.gz
mv hadoop-3.4.1/ hadoop/
cd hadoop/
mkdir -p ~/hadoopdata/hdfs/{namenode,datanode}

Modify ~/.bashrc & $HADOOP_HOME/etc/hadoop/hadoop-env.sh:

cp /etc/bash.bashrc ~/.bashrc # Reset .bashrc file
chown $USER ~/.bashrc # Change ownership of .bashrc file

echo -e "\n#JAVA+HADOOP CONFIG FROM KSKA GIT\nexport JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64\nexport PATH=\$JAVA_HOME/bin:\$PATH" >> ~/.bashrc # Java env var

echo -e "export HADOOP_HOME=/home/hadoop/hadoop/\nexport HADOOP_INSTALL=\$HADOOP_HOME\nexport HADOOP_MAPRED_HOME=\$HADOOP_HOME\nexport HADOOP_COMMON_HOME=\$HADOOP_HOME\nexport HADOOP_HDFS_HOME=\$HADOOP_HOME\nexport YARN_HOME=\$HADOOP_HOME\nexport HADOOP_COMMON_LIB_NATIVE_DIR=\$HADOOP_HOME/lib/native\nexport PATH=\$PATH:\$HADOOP_HOME/sbin:\$HADOOP_HOME/bin\nexport HADOOP_OPTS=\"-Djava.library.path=\$HADOOP_HOME/lib/native\"" >> ~/.bashrc # Hadoop env var

echo -e "PATH=\$PATH:\$HADOOP_HOME/sbin" >> ~/.bashrc

source ~/.bashrc

sed -i 's|^# export JAVA_HOME=.*|export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64/|' "$HADOOP_HOME/etc/hadoop/hadoop-env.sh"

Modify hadoop config files:

sed -i '/<configuration>/,/<\/configuration>/d' $HADOOP_HOME/etc/hadoop/core-site.xml
echo "<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/hadoop/tmp</value>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:9000</value>
</property>
</configuration>" >> $HADOOP_HOME/etc/hadoop/core-site.xml

sed -i '/<configuration>/,/<\/configuration>/d' $HADOOP_HOME/etc/hadoop/hdfs-site.xml
echo "<configuration>
<property>
  <name>dfs.namenode.dir</name>
  <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
</configuration>" >> $HADOOP_HOME/etc/hadoop/hdfs-site.xml

sed -i '/<configuration>/,/<\/configuration>/d' $HADOOP_HOME/etc/hadoop/mapred-site.xml
echo "<configuration>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_HOME/home/hadoop/hadoop/bin/hadoop</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_HOME/home/hadoop/hadoop/bin/hadoop</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_HOME/home/hadoop/hadoop/bin/hadoop</value>
</property>
</configuration>" >> $HADOOP_HOME/etc/hadoop/mapred-site.xml

sed -i '/<configuration>/,/<\/configuration>/d' $HADOOP_HOME/etc/hadoop/yarn-site.xml
echo "<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>localhost</value>
</property>
</configuration>" >> $HADOOP_HOME/etc/hadoop/yarn-site.xml

Format HDFS namenode:

hdfs namenode -format # Format hdfs namenode

Start hadoop cluster:

start-all.sh
jps

Note

Visit localhost:9870 on your browser!

Manually start/stop services

Start all services

# start
hdfs --daemon start namenode
hdfs --daemon start datanode
yarn --daemon start nodemanager
yarn --daemon start resourcemanager
hdfs --daemon start secondarynamenode
hdfs dfsadmin -report
yarn node -list
jps # Check status

Stop all services

# stop
hdfs --daemon stop namenode
hdfs --daemon stop datanode
yarn --daemon stop resourcemanager
yarn --daemon stop nodemanager
hdfs --daemon stop secondarynamenode
jps # Check status

5.4 KiB Executable File Raw Permalink Blame History

Setup for Hadoop

Prerequisites

Manually start/stop services

Start all services

Stop all services

References

5.4 KiB

Executable File

Raw Permalink Blame History