Sqoop Installation

Sqoop Installation on Dataproc cluster


Prerequisite

  • HDFS home directory should be setup before Sqoop installation Access HDFS in Terminal

Download Sqoop

For this Sqoop Installation tutorial, we are using version 1.4.7, that is, sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz For other versions of Sqoop you can follow this link

In the terminal do wget to download the sqoop package.

wget http://apachemirror.wuchna.com/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

Extract the Sqoop tarball and move it to the “/usr/lib/” directory.

tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
sudo mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop
sudo mv sqoop /usr/lib/

Configuring profile

Use vi command to append the following lines to ~/.profile file.

#Sqoop
export SQOOP_HOME=/usr/lib/sqoop 
export PATH=$PATH:$SQOOP_HOME/bin

Now, save and close the above file by executing “:wq!” command.

Execute below command to reload enviroment variables.

source ~/.profile

Configuring Sqoop

While, we need to edit the sqoop-env.sh file, that is placed in the $SQOOP_HOME/conf directory, in order to configure Sqoop with Hadoop. Now, using the following command redirect to Sqoop config directory and copy the template file.

cd /usr/lib/sqoop/conf
mv sqoop-env-template.sh sqoop-env.sh

Also, open sqoop-env.sh and edit the following lines

export HADOOP_COMMON_HOME=/usr/lib/hadoop
export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

Now, save the above file by executing “:wq!” command.

Copy hive-site.xml file into sqoop conf

sudo cp /usr/lib/hive/conf/hive-site.xml /usr/lib/sqoop/conf

Configure MySQL-connector-java

You will find mysql-connector file in /usr/share/java/

You need to set softlink of MySQL-connector in $SQOOP_HOME/lib

sudo ln -s /usr/share/java/mysql-connector-java-5.1.42.jar /usr/lib/sqoop/lib

Verifying Sqoop

For verifying the Sqoop version we use the following command.

cd /usr/lib/sqoop/
./bin/sqoop-version


Reset your mysql or mariadb root password

Stopping the Database Server

To change the root password, you have to shut down the database server beforehand. You can do that for MySQL with:

sudo systemctl stop mysql

After the database server is stopped, you’ll access it manually to reset the root password. Restarting the Database Server Without Permission Checking Start the database without loading the grant tables or enabling networking:

sudo mysqld_safe --skip-grant-tables &

Note: if you face below error then run the below command then execute the above command.

mysqld_safe Directory '/var/run/mysqld' for UNIX socket file don't exists.

Creating the directory manually and setting permissions for it:

sudo mkdir -p /var/run/mysqld

sudo chown mysql:mysql /var/run/mysqld

Now you can connect to the database as the root user, which should not ask for a password.

mysql -u root

You’ll immediately see a database shell prompt instead. Change the Root Password.. Let’s tell the database server to reload the grant tables by issuing the FLUSH PRIVILEGES command.

FLUSH PRIVILEGES;

Now we can actually change the root password. For MySQL 5.7.6 and newer as well as MariaDB 10.1.20 and newer, use the following command.

ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';

Reload the grant tables after this.

FLUSH PRIVILEGES;

Restart the Database Server Normally First, stop the instance of the database server that you started manually

ps aux | grep mysqld

Kill mysql services using kill -9 <process id> command

Then, restart the service using systemctl.

sudo systemctl restart mysql

It gives error as ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) so for restarting the mysql first of all kill the currently running mysql and mysqld demon,

sudo killall -9 mysql

sudo killall -9 mysqld

sudo killall -9 mysqld_safe

sudo /etc/init.d/mysql stop

sudo service mysql start

Now you can confirm that the new password has been applied correctly by running:

mysql -u root -p

Discussion

0 comments