hbase, zookeeper, sqoop, mysql Installation on Windows WSL 2 on Ubuntu 20.04 LTS (Single Node)
This is a follow up of my previous post where Hadoop was installed. If not done yet, please follow previous post.
Install Zookeeper:
Create Zookeeper data directory:
sudo mkdir -p /data/zookeeper
Attach ownership to user ‘hadoop’ on this directory:
sudo chown -R hadoop:hadoop /data/zookeeper
Download and extract Zookeeper tar file and move to /usr/local:
wget https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gztar -xzf hbase-1.7.tar -xvf apache-zookeeper-3.6.3-bin.tar.gz1-bin.tar.gzsudo mv apache-zookeeper-3.6.3-bin zookeepersudo mv zookeeper /usr/local
Configure ZooKeeper in standalone mode. Ccreate a new zoo.cfg file in the zookeeper directory under /usr/local/zookeeper:
sudo nano /opt/zookeeper/conf/zoo.cfg
Add following lines (save and close):
tickTime = 2000
dataDir = /data/zookeeper
clientPort = 2181
initLimit = 5
syncLimit = 2
Start Zookeeper service using below command (it should show like this):
sudo /usr/local/zookeeper/bin/zkServer.sh start
Connect to Zookeeper server on this localhost machine (looks like this):
sudo /usr/local/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181
Type ‘help’ on command prompt to check for help section on Zookeeper commands and ‘quit’ to return back.
Shut down server:
/usr/local/zookeeper/bin/zkServer.sh stop
Download and install Hive 1.7.1, extract and move folder to /usr/local:
wget https://dlcdn.apache.org/hbase/1.7.1/hbase-1.7.1-bin.tar.gztar -xzf hbase-1.7.1-bin.tar.gz
sudo mv hbase-1.7.1 HBase
sudo mv HBase /usr/local
sudo chmod 777 /usr/local/HBase
Open .bashrc file
code ~/.bashrc OR sudo nano ~/.bashrc
and add below lines in end (save and close):
export HBASE_HOME=/usr/local/HBase
export PATH=$PATH:$HBASE_HOME/bin
Open hbase-env.sh
code /usr/local/HBase/conf/hbase-env.sh
OR
sudo nano /usr/local/HBase/conf/hbase-env.sh
Add these lines in end (to know JAVA_HOME path, use command $JAVA_HOME…save and close):
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
Edit this file
code /usr/local/HBase/conf/hbase-site.xml
OR
sudo nano /usr/local/HBase/conf/hbase-site.xml
add following lines to make configuration look like:
<configuration>
<property><name>hbase.rootdir</name><value>hdfs://localhost:8030/hbase</value></property><property><name>hbase.zookeeper.property.dataDir</name><value>/data/zookeeper</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property>
</configuration>
Before running HBase, make sure HDFS is running using jps command (if not then ‘start-all.sh’ under hadoop/bin folder should do it:
jps
Once confirmed then start HBase:
sudo /usr/local/HBase/bin/start-hbase.sh
Enter HBase shell (looks like this):
sudo /usr/local/HBase/bin/hbase shell
Type “help” to get list of supported commands and type “exit” to quit this interactive shell.
Sqoop Installation:
Download Sqoop and extract and move to /usr/local:
wget http://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gztar -xzf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gzsudo mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoopsudo mv sqoop /usr/local
Open .bashrc file:
code ~/.bashrc
Add following lines in end (save and close):
export SQOOP_HOME=/usr/local/sqoopexport PATH=$PATH:$SQOOP_HOME/bin
Reload bashrc file:
source ~/.bashrc
Configure sqoop:
cd $SQOOP_HOME/confmv sqoop-env-template.sh sqoop-env.sh
Open file:
code sqoop-env.sh
OR
sudo nano sqoop-env.sh
add these lines in end (Save and close):
export HADOOP_COMMON_HOME=/usr/local/hadoopexport HADOOP_MAPRED_HOME=/usr/local/hadoop
Configure SQL connector for sqoop:
cd ~wget http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gztar -xzf mysql-connector-java-5.1.48.tar.gzcd mysql-connector-java-5.1.48/
Move the bin.jar file in sqoop/lib:
sudo mv mysql-connector-java-5.1.48-bin.jar /usr/local/sqoop/libcd ~
Check sqoop status (should show like this):
cd $SQOOP_HOME/binsqoop version
Install MYSQL:
Download MySQL package and instll package using dpkg:
cd ~wget https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.debdpkg -i mysql-apt-config_0.8.20-1_all.deb
Press enter to see next page, press enter to keep same selection as shown below:
Traverse down to ok and enter. Run next commands:
sudo apt updatesudo apt install mysql-server mysql-client -ysudo mysql_secure_installation
Some times you dont get the expected result from above command which is :
In that case restart mysql services and run previous step:
sudo service mysql stopsudo service mysql startsudo mysql_secure_installation
For ‘VALIDATE PASSWORD COMPONENT’ I have selected no.
Enter password and re-enter password.
Remove anonymous users : Y
Disallow root login remotely: y
Remove test database and access to it: y
Reload privilege tables now: y
Login to MySQL server (type password when prompted and prompt will change o mysql…should look like below):
sudo mysql -u root -p
Test SQL connection with Sqoop(create database/table and sqoop to hdfs):
create database tutorial;show databases;use tutorial;create table student(
rno int AUTO_INCREMENT PRIMARY KEY,
name varchar(20),
marks int,
grade char(2));show tables;desc student;
Insert data into table:
insert into student values (1,'Ravi',80,'A');
insert into student values (2,'John',20,'D');
insert into student values (3,'Michel',70,'A');
insert into student values (4,'Grah',60,'B');
insert into student values (5,'Sunny',40,'D');
insert into student values (6,'David',90,'A');
insert into student values (7,'Peter',75,'A');
insert into student values (8,'Jack',66,'B');
insert into student values (9,'James',69,'B');
insert into student values (10,'Roman',75,'A');
insert into student values (11,'Mia',95,'A');
insert into student values (12,'Kiri',30,'D');
insert into student values (13,'johan',80,'A');
insert into student values (14,'Maha',88,'A');
insert into student values (15,'Julius',64,'B');
insert into student values (16,'Thompson',86,'A');
insert into student values (17,'William',76,'A');
insert into student values (18,'Jacques',57,'C');
insert into student values (19,'Harris',76,'A');
insert into student values (20,'Danira',33,'D');
insert into student values (7,'Peter',75,'A');
Check table:
select * from student;
Create a super user on ‘hadoop’ on mysql :
sudo mysql -u root -pCREATE USER 'hadoop'@'localhost' IDENTIFIED BY 'hadoop';GRANT ALL PRIVILEGES ON *.* TO 'hadoop'@'localhost' WITH GRANT OPTION;CREATE USER 'hadoop'@'%' IDENTIFIED BY 'hadoop';GRANT ALL PRIVILEGES ON *.* TO 'hadoop'@'%' WITH GRANT OPTION;FLUSH PRIVILEGES;exit
Test sqoop command to connect with MYSQL (result should look like this):
sqoop list-databases --connect jdbc:mysql://localhost/ --username hadoop -P
Sqoop table data from MySQL:
sqoop eval --connect jdbc:mysql://localhost/tutorial --username hadoop --password hadoop --query "select * from student"
Sqoop table data from MySQL into HDFS (check on HDFS by command):
hdfs dfs -ls /user/test_import/student
So we have successfully installed Sqoop, MySQL, HBase, Zookeper and Sqooped data from MySQL to HDFS. Thank you !