sqoop基于Hadoop与Hive

Hadoop https://www.cnblogs.com/xibuhaohao/p/11772031.html  

Hive      https://www.cnblogs.com/xibuhaohao/p/11772481.html

回到顶部

一、下载sqoop

sqoop-1.4.7.tar 

http://mirror.bit.edu.cn/apache/sqoop/1.4.7/

回到顶部

二、安装sqoop

 

1、解压缩sqoop

tar -vzxf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /home/hadoop/

 

 

2、sqoop文件目录重命名 

mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7.bin

 

3、配置sqoop环境变量

export SQOOP_HOME=/home/hadoop/sqoop-1.4.7.bin
export PATH=$PATH:$JAVA_HOME/bin:$SQOOP_HOME/bin

回到顶部

三、安装配置sqoop

 

1、配置sqoop配置文件

cd /home/hadoop/sqoop-1.4.7.bin/conf

cp sqoop-env-template.sh sqoop-env.sh

cat sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.8.5
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.8.5
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/apache-hive-2.3.6-bin
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

 

2、验证安装配置

  
回到顶部

四、sqoop测试

 

1、MySQL测试

1）mysql-connector-java-5.1.48-bin.jar放入/hadoop/sqoop-1.4.7/lib目录下
cp mysql-connector-java-5.1.48.jar /home/hadoop/sqoop-1.4.7.bin//lib/
2）sqoop连接MySQL测试
进入/hadoop/sqoop-1.4.7/bin 运行（已经设置环境变量则可以在任意目录运行）
sqoop list-databases –connect jdbc:mysql://172.16.100.173:3306/ –username root –P
 

3）将MySQL数据导入Hive

sqoop import –connect jdbc:mysql://172.16.100.173:3306/hdb –username root –password oracletest –table htest –fields-terminated-by ‘\t’ –delete-target-dir –num-mappers 1 –hive-import –hive-database hdb –hive-table htest

回到顶部

五、问题处理

1、ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

环境变量中加如

export HIVE_CONF_DIR=/home/hadoop/apache-hive-2.3.6-bin/conf
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*

2、ERROR Could not register mbeans java.security.AccessControlException: access denied (“javax.management.MBeanTrustPermission” “register”)

解决办法: 

vim $JAVA_HOME/jre/lib/security/java.policy

在grant{}内部添加如下内容：

permission javax.management.MBeanTrustPermission “register”;

3、ERROR exec.TaskRunner: Error in executeTask

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader

4、ERROR tool.ImportTool: Import failed: java.io.IOException: Hive CliDriver exited with status=-101

5、FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MoveTask. com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;
19/11/01 14:24:14 ERROR ql.Driver: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.MoveTask. com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;

3、4、5都通过将hive $HIVE_HOME/lib/jackson*.jar 拷贝至 $SQOOP_HOME/lib 目录中，解决

将$SQOOP_HOME/lib/jackson*.jar 文件bak，再把$HIVE_HOME/lib/jackson*.jar 拷贝至 $SQOOP_HOME/lib 目录中，重新运行sqoop 作业，导入成功。