Sqoop Installation on Dataproc cluster
- HDFS home directory should be setup before Sqoop installation Access HDFS in Terminal
For this Sqoop Installation tutorial, we are using version 1.4.7, that is, sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz For other versions of Sqoop you can follow this link
In the terminal do wget to download the sqoop package.
Extract the Sqoop tarball and move it to the “/usr/lib/” directory.
tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz sudo mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop sudo mv sqoop /usr/lib/
Use vi command to append the following lines to ~/.profile file.
#Sqoop export SQOOP_HOME=/usr/lib/sqoop export PATH=$PATH:$SQOOP_HOME/bin
Now, save and close the above file by executing “:wq!” command.
Execute below command to reload enviroment variables.
While, we need to edit the sqoop-env.sh file, that is placed in the $SQOOP_HOME/conf directory, in order to configure Sqoop with Hadoop. Now, using the following command redirect to Sqoop config directory and copy the template file.
cd /usr/lib/sqoop/conf mv sqoop-env-template.sh sqoop-env.sh
Also, open sqoop-env.sh and edit the following lines
export HADOOP_COMMON_HOME=/usr/lib/hadoop export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
Now, save the above file by executing “:wq!” command.
Copy hive-site.xml file into sqoop conf
sudo cp /usr/lib/hive/conf/hive-site.xml /usr/lib/sqoop/conf
You will find mysql-connector file in /usr/share/java/
You need to set softlink of MySQL-connector in $SQOOP_HOME/lib
sudo ln -s /usr/share/java/mysql-connector-java-5.1.42.jar /usr/lib/sqoop/lib
For verifying the Sqoop version we use the following command.
cd /usr/lib/sqoop/ ./bin/sqoop-version
Reset your mysql or mariadb root password
Stopping the Database Server
To change the root password, you have to shut down the database server beforehand. You can do that for MySQL with:
sudo systemctl stop mysql
After the database server is stopped, you’ll access it manually to reset the root password. Restarting the Database Server Without Permission Checking Start the database without loading the grant tables or enabling networking:
sudo mysqld_safe --skip-grant-tables &
Note: if you face below error then run the below command then execute the above command.
mysqld_safe Directory '/var/run/mysqld' for UNIX socket file don't exists.
Creating the directory manually and setting permissions for it:
sudo mkdir -p /var/run/mysqld
sudo chown mysql:mysql /var/run/mysqld
Now you can connect to the database as the root user, which should not ask for a password.
mysql -u root
You’ll immediately see a database shell prompt instead. Change the Root Password.. Let’s tell the database server to reload the grant tables by issuing the FLUSH PRIVILEGES command.
Now we can actually change the root password. For MySQL 5.7.6 and newer as well as MariaDB 10.1.20 and newer, use the following command.
ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';
Reload the grant tables after this.
Restart the Database Server Normally First, stop the instance of the database server that you started manually
ps aux | grep mysqld
Kill mysql services using kill -9 <process id> command
Then, restart the service using systemctl.
sudo systemctl restart mysql
It gives error as ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) so for restarting the mysql first of all kill the currently running mysql and mysqld demon,
sudo killall -9 mysql
sudo killall -9 mysqld
sudo killall -9 mysqld_safe
sudo /etc/init.d/mysql stop
sudo service mysql start
Now you can confirm that the new password has been applied correctly by running:
mysql -u root -p