java操作sqoop在本地测试正常打包发布到阿里云服务器报错

在做项目时，把sqoop和mapreduce集成到springboot中做数据抽取和计算，并打成jar包上传到服务器，项目是使用eureka进行注册提供服务，父项目下创建子模块，通过maven的package打包子模块。
通过以下命令在服务器运行

# nohup和&组合在后台运行 运行时会在当前目录下生成一个nohup.out日志文件
[root@fda ~]# nohup java -jar XXX.jar &
 
Sqoop版本1.4.7
Hadoop版本2.7.7
当进行sqoop的数据抽取时报错信息如下

2021-02-27 19:24 | [34mINFO [0;39m | [1;33mtask-1[0;39m | [1;32mo.a.sqoop.orm.CompilationManager[0;39m | $HADOOP_MAPRED_HOME is not set
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:10: error: package org.apache.hadoop.mapred.lib.db does not exist
import org.apache.hadoop.mapred.lib.db.DBWritable;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:11: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.JdbcWritableBridge;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:12: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.DelimiterSet;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:13: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.FieldFormatter;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:14: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.RecordParser;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:15: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.BooleanParser;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:16: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.BlobRef;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:17: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.ClobRef;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:18: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.LargeObjectLoader;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:19: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.SqoopRecord;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:37: error: cannot find symbol
public class Users extends SqoopRecord implements DBWritable, Writable {
^
symbol: class SqoopRecord
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:37: error: cannot find symbol
public class Users extends SqoopRecord implements DBWritable, Writable {
^
symbol: class DBWritable
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:37: error: cannot find symbol
public class Users extends SqoopRecord implements DBWritable, Writable {
^
symbol: class Writable
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:137: error: cannot find symbol
public void loadLargeObjects(LargeObjectLoader __loader)
^
symbol: class LargeObjectLoader
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:140: error: cannot find symbol
public void loadLargeObjects0(LargeObjectLoader __loader)
^
symbol: class LargeObjectLoader
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:217: error: cannot find symbol
private static final DelimiterSet __outputDelimiters = new DelimiterSet((char) 9, (char) 10, (char) 0, (char) 0, false);
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:221: error: cannot find symbol
public String toString(DelimiterSet delimiters) {
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:227: error: cannot find symbol
public String toString(DelimiterSet delimiters, boolean useRecordDelim) {
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:240: error: cannot find symbol
public void toString0(DelimiterSet delimiters, StringBuilder __sb, char fieldDelim) {
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:247: error: cannot find symbol
private static final DelimiterSet __inputDelimiters = new DelimiterSet((char) 9, (char) 10, (char) 0, (char) 0, false);
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:248: error: cannot find symbol
private RecordParser __parser;
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:249: error: cannot find symbol
public void parse(Text __record) throws RecordParser.ParseError {
^
symbol: class Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:249: error: package RecordParser does not exist
public void parse(Text __record) throws RecordParser.ParseError {
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:257: error: package RecordParser does not exist
public void parse(CharSequence __record) throws RecordParser.ParseError {
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:265: error: package RecordParser does not exist
public void parse(byte [] __record) throws RecordParser.ParseError {
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:273: error: package RecordParser does not exist
public void parse(char [] __record) throws RecordParser.ParseError {
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:281: error: package RecordParser does not exist
public void parse(ByteBuffer __record) throws RecordParser.ParseError {
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:289: error: package RecordParser does not exist
public void parse(CharBuffer __record) throws RecordParser.ParseError {
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:128: error: cannot find symbol
this.mobile = JdbcWritableBridge.readString(1, __dbResults);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:129: error: cannot find symbol
this.userName = JdbcWritableBridge.readString(2, __dbResults);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:130: error: cannot find symbol
this.password = JdbcWritableBridge.readString(3, __dbResults);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:133: error: cannot find symbol
this.mobile = JdbcWritableBridge.readString(1, __dbResults);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:134: error: cannot find symbol
this.userName = JdbcWritableBridge.readString(2, __dbResults);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:135: error: cannot find symbol
this.password = JdbcWritableBridge.readString(3, __dbResults);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:148: error: cannot find symbol
JdbcWritableBridge.writeString(mobile, 1 + __off, 12, __dbStmt);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:149: error: cannot find symbol
JdbcWritableBridge.writeString(userName, 2 + __off, 12, __dbStmt);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:150: error: cannot find symbol
JdbcWritableBridge.writeString(password, 3 + __off, 12, __dbStmt);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:154: error: cannot find symbol
JdbcWritableBridge.writeString(mobile, 1 + __off, 12, __dbStmt);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:155: error: cannot find symbol
JdbcWritableBridge.writeString(userName, 2 + __off, 12, __dbStmt);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:156: error: cannot find symbol
JdbcWritableBridge.writeString(password, 3 + __off, 12, __dbStmt);
^
symbol: variable JdbcWritableBridge
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:164: error: cannot find symbol
this.mobile = Text.readString(__dataIn);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:169: error: cannot find symbol
this.userName = Text.readString(__dataIn);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:174: error: cannot find symbol
this.password = Text.readString(__dataIn);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:182: error: cannot find symbol
Text.writeString(__dataOut, mobile);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:188: error: cannot find symbol
Text.writeString(__dataOut, userName);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:194: error: cannot find symbol
Text.writeString(__dataOut, password);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:202: error: cannot find symbol
Text.writeString(__dataOut, mobile);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:208: error: cannot find symbol
Text.writeString(__dataOut, userName);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:214: error: cannot find symbol
Text.writeString(__dataOut, password);
^
symbol: variable Text
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:217: error: cannot find symbol
private static final DelimiterSet __outputDelimiters = new DelimiterSet((char) 9, (char) 10, (char) 0, (char) 0, false);
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:230: error: cannot find symbol
__sb.append(FieldFormatter.escapeAndEnclose(mobile==null?”\\N”:mobile, delimiters));
^
symbol: variable FieldFormatter
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:232: error: cannot find symbol
__sb.append(FieldFormatter.escapeAndEnclose(userName==null?”\\N”:userName, delimiters));
^
symbol: variable FieldFormatter
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:234: error: cannot find symbol
__sb.append(FieldFormatter.escapeAndEnclose(password==null?”\\N”:password, delimiters));
^
symbol: variable FieldFormatter
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:241: error: cannot find symbol
__sb.append(FieldFormatter.escapeAndEnclose(mobile==null?”\\N”:mobile, delimiters));
^
symbol: variable FieldFormatter
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:243: error: cannot find symbol
__sb.append(FieldFormatter.escapeAndEnclose(userName==null?”\\N”:userName, delimiters));
^
symbol: variable FieldFormatter
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:245: error: cannot find symbol
__sb.append(FieldFormatter.escapeAndEnclose(password==null?”\\N”:password, delimiters));
^
symbol: variable FieldFormatter
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:247: error: cannot find symbol
private static final DelimiterSet __inputDelimiters = new DelimiterSet((char) 9, (char) 10, (char) 0, (char) 0, false);
^
symbol: class DelimiterSet
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:251: error: cannot find symbol
this.__parser = new RecordParser(__inputDelimiters);
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:259: error: cannot find symbol
this.__parser = new RecordParser(__inputDelimiters);
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:267: error: cannot find symbol
this.__parser = new RecordParser(__inputDelimiters);
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:275: error: cannot find symbol
this.__parser = new RecordParser(__inputDelimiters);
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:283: error: cannot find symbol
this.__parser = new RecordParser(__inputDelimiters);
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:291: error: cannot find symbol
this.__parser = new RecordParser(__inputDelimiters);
^
symbol: class RecordParser
location: class Users
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:363: error: cannot find symbol
Users o = (Users) super.clone();
^
symbol: variable super
location: class Users
67 errors
2021-02-27 19:24 | [1;31mERROR[0;39m | [1;33mtask-1[0;39m | [1;32morg.apache.sqoop.tool.ImportTool[0;39m | Import failed: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:226)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at zut.edu.service.impl.DataExtractionServiceImpl.sqoopTask(DataExtractionServiceImpl.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
 
这里一共有两个问题第一个问题

/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;
 
这是由于阿里云的环境变量没有配置完整Hadoop的环境变量。在服务器的 /etc/profile 中添加如下配置
#Hadoop环境变量

[root@fda ~]# vim /etc/profile
#添加下面的内容
export HADOOP_HOME=/opt/module/Hadoop/hadoop-2.7.7
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export YARN_HOME=$HADOOP_HOME
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_LOG_DIR=$HADOOP_LOG_DIR
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin 
第一个问题解决。
第二个问题是

/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:11: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.JdbcWritableBridge;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:12: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.DelimiterSet;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:13: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.FieldFormatter;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:14: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.RecordParser;
^
/tmp/sqoop-root/compile/f677c4605f731efc068e40b146260436/Users.java:15: error: package com.cloudera.sqoop.lib does not exist
import com.cloudera.sqoop.lib.BooleanParser; 
报的sqoop的找不到sqoop-1.4.7.jar下面的类，打开打成的可执行jar包，在BOOT-INF\lib下确实有sqoop-1.4.7.jar，找了好几天最终发现问题，是因为打包的原因，但是又找不到怎样去打包，索性直接把原来的后台项目架构改成了单体的springboot项目，打成war包部署到服务器，问题解决。

原来打包问题待进一步研究……如果小伙伴有解决方案，请与我分享一下，感谢！！！

————————————————
版权声明：本文为CSDN博主「hykDatabases」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/weixin_45102492/article/details/114355799