hbase, zookeeper, sqoop, mysql Installation on Windows WSL 2 on Ubuntu 20.04 LTS (Single Node)

This is a follow up of my previous post where Hadoop was installed. If not done yet, please follow previous post.

Install Zookeeper:

Create Zookeeper data directory:

sudo mkdir -p /data/zookeeper

Attach ownership to user ‘hadoop’ on this directory:

sudo chown -R hadoop:hadoop /data/zookeeper

Download and extract Zookeeper tar file and move to /usr/local:

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz
tar -xzf hbase-1.7.tar -xvf apache-zookeeper-3.6.3-bin.tar.gz1-bin.tar.gz
sudo mv apache-zookeeper-3.6.3-bin zookeeper
sudo mv zookeeper /usr/local

Configure ZooKeeper in standalone mode. Ccreate a new zoo.cfg file in the zookeeper directory under /usr/local/zookeeper:

sudo nano /opt/zookeeper/conf/zoo.cfg

Add following lines (save and close):

tickTime = 2000  
dataDir = /data/zookeeper  
clientPort = 2181  
initLimit = 5  
syncLimit = 2

Start Zookeeper service using below command (it should show like this):

sudo /usr/local/zookeeper/bin/zkServer.sh start

Connect to Zookeeper server on this localhost machine (looks like this):

sudo /usr/local/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181

Type ‘help’ on command prompt to check for help section on Zookeeper commands and ‘quit’ to return back.

Shut down server:

/usr/local/zookeeper/bin/zkServer.sh stop

Download and install Hive 1.7.1, extract and move folder to /usr/local:

wget https://dlcdn.apache.org/hbase/1.7.1/hbase-1.7.1-bin.tar.gz
tar -xzf hbase-1.7.1-bin.tar.gz
sudo mv hbase-1.7.1 HBase
sudo mv HBase /usr/local
sudo chmod 777 /usr/local/HBase

Open .bashrc file

code ~/.bashrc OR sudo nano ~/.bashrc

and add below lines in end (save and close):

export HBASE_HOME=/usr/local/HBase
export PATH=$PATH:$HBASE_HOME/bin

Open hbase-env.sh

code /usr/local/HBase/conf/hbase-env.sh
OR 
sudo nano /usr/local/HBase/conf/hbase-env.sh

Add these lines in end (to know JAVA_HOME path, use command $JAVA_HOME…save and close):

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

Edit this file

code /usr/local/HBase/conf/hbase-site.xml
OR 
sudo nano /usr/local/HBase/conf/hbase-site.xml

add following lines to make configuration look like:

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:8030/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/data/zookeeper</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>

Before running HBase, make sure HDFS is running using jps command (if not then ‘start-all.sh’ under hadoop/bin folder should do it:

jps

Once confirmed then start HBase:

sudo /usr/local/HBase/bin/start-hbase.sh

Enter HBase shell (looks like this):

sudo /usr/local/HBase/bin/hbase shell

Type “help” to get list of supported commands and type “exit” to quit this interactive shell.

Sqoop Installation:

Download Sqoop and extract and move to /usr/local:

wget http://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
tar -xzf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
sudo mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop
sudo mv sqoop /usr/local

Open .bashrc file:

code ~/.bashrc

Add following lines in end (save and close):

export SQOOP_HOME=/usr/local/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

Reload bashrc file:

source ~/.bashrc

Configure sqoop:

cd $SQOOP_HOME/conf
mv sqoop-env-template.sh sqoop-env.sh

Open file:

code sqoop-env.sh
OR 
sudo nano sqoop-env.sh

add these lines in end (Save and close):

export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop

Configure SQL connector for sqoop:

cd ~
wget http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz
tar -xzf mysql-connector-java-5.1.48.tar.gz
cd mysql-connector-java-5.1.48/

Move the bin.jar file in sqoop/lib:

sudo mv mysql-connector-java-5.1.48-bin.jar /usr/local/sqoop/lib
cd ~

Check sqoop status (should show like this):

cd $SQOOP_HOME/bin
sqoop version

Install MYSQL:

Download MySQL package and instll package using dpkg:

cd ~
wget https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
dpkg -i mysql-apt-config_0.8.20-1_all.deb

Press enter to see next page, press enter to keep same selection as shown below:

Traverse down to ok and enter. Run next commands:

sudo apt update
sudo apt install mysql-server mysql-client -y
sudo mysql_secure_installation

Some times you dont get the expected result from above command which is :

In that case restart mysql services and run previous step:

sudo service mysql stop
sudo service mysql start
sudo mysql_secure_installation

For ‘VALIDATE PASSWORD COMPONENT’ I have selected no.
Enter password and re-enter password.
Remove anonymous users : Y
Disallow root login remotely: y
Remove test database and access to it: y
Reload privilege tables now: y

Login to MySQL server (type password when prompted and prompt will change o mysql…should look like below):

sudo mysql -u root -p

Test SQL connection with Sqoop(create database/table and sqoop to hdfs):

create database tutorial;
show databases;
use tutorial;
create table student(
rno int AUTO_INCREMENT PRIMARY KEY,
name varchar(20),
marks int,
grade char(2));
show tables;
desc student;

Insert data into table:

insert into student values (1,'Ravi',80,'A');
insert into student values (2,'John',20,'D');
insert into student values (3,'Michel',70,'A');
insert into student values (4,'Grah',60,'B');
insert into student values (5,'Sunny',40,'D');
insert into student values (6,'David',90,'A');
insert into student values (7,'Peter',75,'A');
insert into student values (8,'Jack',66,'B');
insert into student values (9,'James',69,'B');
insert into student values (10,'Roman',75,'A');
insert into student values (11,'Mia',95,'A');
insert into student values (12,'Kiri',30,'D');
insert into student values (13,'johan',80,'A');
insert into student values (14,'Maha',88,'A');
insert into student values (15,'Julius',64,'B');
insert into student values (16,'Thompson',86,'A');
insert into student values (17,'William',76,'A');
insert into student values (18,'Jacques',57,'C');
insert into student values (19,'Harris',76,'A');
insert into student values (20,'Danira',33,'D');
insert into student values (7,'Peter',75,'A');

Check table:

select * from student;

Create a super user on ‘hadoop’ on mysql :

sudo mysql -u root -p
CREATE USER 'hadoop'@'localhost' IDENTIFIED BY 'hadoop';
GRANT ALL PRIVILEGES ON *.* TO 'hadoop'@'localhost' WITH GRANT OPTION;
CREATE USER 'hadoop'@'%' IDENTIFIED BY 'hadoop';
GRANT ALL PRIVILEGES ON *.* TO 'hadoop'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
exit

Test sqoop command to connect with MYSQL (result should look like this):

sqoop list-databases --connect jdbc:mysql://localhost/ --username hadoop -P

Sqoop table data from MySQL:

sqoop eval --connect jdbc:mysql://localhost/tutorial --username hadoop --password hadoop --query "select * from student"

Sqoop table data from MySQL into HDFS (check on HDFS by command):

hdfs dfs -ls /user/test_import/student

So we have successfully installed Sqoop, MySQL, HBase, Zookeper and Sqooped data from MySQL to HDFS. Thank you !