使用Sqoop将数据从Hive导入MySQL报错（厦大林子雨的基于Hive和Spark的淘宝双11数据分析与预测）
bin/sqoop export --connect jdbc:mysql://localhost:3306/dbtaobao --username root --password root --table user_log --export-dir '/user/hive/warehouse/dbtaobao.db/inner_user_log' --fields-terminated-by ',';
Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
/usr/local/hadoop/libexec/hadoop-functions.sh: 行 2360: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER：无效的变量名
/usr/local/hadoop/libexec/hadoop-functions.sh: 行 2455: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS：无效的变量名
错误: 找不到或无法加载主类 org.apache.hadoop.hbase.util.GetJavaProperty
/usr/local/hadoop/libexec/hadoop-functions.sh: 行 2360: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER：无效的变量名
/usr/local/hadoop/libexec/hadoop-functions.sh: 行 2455: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS：无效的变量名
2023-06-11 19:32:10,965 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
2023-06-11 19:32:10,999 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2023-06-11 19:32:11,070 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2023-06-11 19:32:11,070 INFO tool.CodeGenTool: Beginning code generation
Sun Jun 11 19:32:11 PDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2023-06-11 19:32:11,555 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_log` AS t LIMIT 1
2023-06-11 19:32:11,592 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_log` AS t LIMIT 1
2023-06-11 19:32:11,603 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
注: /tmp/sqoop-hadoop/compile/221ca2ac8b969ea2014308ec2f343c08/user_log.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
2023-06-11 19:32:12,673 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/221ca2ac8b969ea2014308ec2f343c08/user_log.jar
2023-06-11 19:32:12,685 INFO mapreduce.ExportJobBase: Beginning export of user_log
2023-06-11 19:32:12,685 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2023-06-11 19:32:12,778 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2023-06-11 19:32:13,296 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-11 19:32:13,366 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
2023-06-11 19:32:13,369 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2023-06-11 19:32:13,370 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2023-06-11 19:32:13,424 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2023-06-11 19:32:13,474 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2023-06-11 19:32:13,474 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2023-06-11 19:32:13,643 INFO input.FileInputFormat: Total input files to process : 1
2023-06-11 19:32:13,647 INFO input.FileInputFormat: Total input files to process : 1
2023-06-11 19:32:13,667 INFO mapreduce.JobSubmitter: number of splits:4
2023-06-11 19:32:13,693 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2023-06-11 19:32:13,757 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1361289283_0001
2023-06-11 19:32:13,757 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-06-11 19:32:13,865 INFO mapred.LocalDistributedCacheManager: Creating symlink: /usr/local/hadoop/tmp/mapred/local/1686537133805/libjars <- /usr/local/sqoop/libjars/*
2023-06-11 19:32:13,867 WARN fs.FileUtil: Command 'ln -s /usr/local/hadoop/tmp/mapred/local/1686537133805/libjars /usr/local/sqoop/libjars/*' failed 1 with: ln: 无法创建符号链接 '/usr/local/sqoop/libjars/*': 没有那个文件或目录
2023-06-11 19:32:13,867 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /usr/local/hadoop/tmp/mapred/local/1686537133805/libjars <- /usr/local/sqoop/libjars/*
2023-06-11 19:32:13,867 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/hadoop1361289283/.staging/job_local1361289283_0001/libjars as file:/usr/local/hadoop/tmp/mapred/local/1686537133805/libjars
2023-06-11 19:32:13,901 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2023-06-11 19:32:13,902 INFO mapreduce.Job: Running job: job_local1361289283_0001
2023-06-11 19:32:13,904 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2023-06-11 19:32:13,907 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.sqoop.mapreduce.NullOutputCommitter
2023-06-11 19:32:13,929 INFO mapred.LocalJobRunner: Waiting for map tasks
2023-06-11 19:32:13,929 INFO mapred.LocalJobRunner: Starting task: attempt_local1361289283_0001_m_000000_0
2023-06-11 19:32:13,964 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2023-06-11 19:32:13,971 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:355065+59179,/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:414244+59179
2023-06-11 19:32:13,979 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
2023-06-11 19:32:13,980 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
2023-06-11 19:32:13,980 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
2023-06-11 19:32:13,988 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Sun Jun 11 19:32:14 PDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2023-06-11 19:32:14,031 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-11 19:32:14,037 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
2023-06-11 19:32:14,047 INFO mapred.LocalJobRunner: Starting task: attempt_local1361289283_0001_m_000001_0
2023-06-11 19:32:14,050 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2023-06-11 19:32:14,050 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:0+118355
2023-06-11 19:32:14,053 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Sun Jun 11 19:32:14 PDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2023-06-11 19:32:14,077 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
2023-06-11 19:32:14,079 INFO mapred.LocalJobRunner: Starting task: attempt_local1361289283_0001_m_000002_0
2023-06-11 19:32:14,082 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2023-06-11 19:32:14,082 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:118355+118355
2023-06-11 19:32:14,087 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Sun Jun 11 19:32:14 PDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2023-06-11 19:32:14,112 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-11 19:32:14,116 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
2023-06-11 19:32:14,118 INFO mapred.LocalJobRunner: Starting task: attempt_local1361289283_0001_m_000003_0
2023-06-11 19:32:14,119 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2023-06-11 19:32:14,120 INFO mapred.MapTask: Processing split: Paths:/user/hive/warehouse/dbtaobao.db/inner_user_log/000000_0:236710+118355
2023-06-11 19:32:14,123 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Sun Jun 11 19:32:14 PDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2023-06-11 19:32:14,152 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-06-11 19:32:14,160 INFO mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
2023-06-11 19:32:14,162 INFO mapred.LocalJobRunner: map task executor complete.
2023-06-11 19:32:14,162 WARN mapred.LocalJobRunner: job_local1361289283_0001
java.lang.Exception: java.io.IOException: java.lang.ClassNotFoundException: user_log
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: java.lang.ClassNotFoundException: user_log
at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:70)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: user_log
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.sqoop.mapreduce.TextExportMapper.setup(TextExportMapper.java:66)
... 10 more
2023-06-11 19:32:14,904 INFO mapreduce.Job: Job job_local1361289283_0001 running in uber mode : false
2023-06-11 19:32:14,905 INFO mapreduce.Job: map 0% reduce 0%
2023-06-11 19:32:14,906 INFO mapreduce.Job: Job job_local1361289283_0001 failed with state FAILED due to: NA
2023-06-11 19:32:14,911 INFO mapreduce.Job: Counters: 0
2023-06-11 19:32:14,913 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2023-06-11 19:32:14,914 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 1.5376 seconds (0 bytes/sec)
2023-06-11 19:32:14,915 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2023-06-11 19:32:14,915 INFO mapreduce.ExportJobBase: Exported 0 records.
2023-06-11 19:32:14,915 ERROR tool.ExportTool: Error during export: Export job failed!