Sqoop執行命令：
./sqoop import --connect jdbc:oracle:thin:@10.112.101.251:1621:crmadbmr --username bass_etl --password 75!n!u6J --table DBPMSADM.PD_USERSVC_INFO_00 -m 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906
執行日誌：
crmd3n:/d2_data0/user/ocdc/bin/sqoop-1.4.2-cdh4.2.1/bin> 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906 <
14/07/19 15:31:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/07/19 15:31:30 INFO manager.SqlManager: Using default fetchSize of 1000
14/07/19 15:31:30 INFO tool.CodeGenTool: Beginning code generation
14/07/19 15:31:31 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DBPMSADM.PD_USERSVC_INFO_00 t WHERE 1=0
14/07/19 15:31:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /d2_data0/user/ocdc/bin/hadoop-2.0.0-mr1-cdh4.2.1
注意：/tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM_PD_USERSVC_INFO_00.java 使用或覆蓋了已過時的 API。
注意：要了解詳細信息，請使用 -Xlint:deprecation 重新編譯。
14/07/19 15:31:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM.PD_USERSVC_INFO_00.jar
14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.
14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.
14/07/19 15:31:33 INFO mapreduce.ImportJobBase: Beginning import of DBPMSADM.PD_USERSVC_INFO_00
14/07/19 15:31:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:34 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/07/19 15:31:36 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(SVC_ID), MAX(SVC_ID) FROM DBPMSADM.PD_USERSVC_INFO_00
14/07/19 15:31:37 WARN db.TextSplitter: Generating splits for a textual index column.
14/07/19 15:31:37 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
14/07/19 15:31:37 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
14/07/19 15:31:38 INFO mapred.JobClient: Running job: job_201407151514_3489
14/07/19 15:31:39 INFO mapred.JobClient: map 0% reduce 0%
14/07/19 15:31:50 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:53 INFO mapred.JobClient: map 25% reduce 0%
14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:59 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:32:05 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:32:06 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:32:08 INFO mapred.JobClient: map 50% reduce 0%
14/07/19 15:32:17 INFO mapred.JobClient: Job complete: job_201407151514_3489
14/07/19 15:32:17 INFO mapred.JobClient: Counters: 24
14/07/19 15:32:17 INFO mapred.JobClient: File System Counters
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes read=0
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes written=369072
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of read operations=0
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of large read operations=0
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of write operations=0
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes read=256
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes written=8
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of read operations=2
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of large read operations=0
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of write operations=2
14/07/19 15:32:17 INFO mapred.JobClient: Job Counters
14/07/19 15:32:17 INFO mapred.JobClient: Failed map tasks=1
14/07/19 15:32:17 INFO mapred.JobClient: Launched map tasks=12
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=91182
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/07/19 15:32:17 INFO mapred.JobClient: Map-Reduce Framework
14/07/19 15:32:17 INFO mapred.JobClient: Map input records=0
14/07/19 15:32:17 INFO mapred.JobClient: Map output records=0
14/07/19 15:32:17 INFO mapred.JobClient: Input split bytes=256
14/07/19 15:32:17 INFO mapred.JobClient: Spilled Records=0
14/07/19 15:32:17 INFO mapred.JobClient: CPU time spent (ms)=8910
14/07/19 15:32:17 INFO mapred.JobClient: Physical memory (bytes) snapshot=698327040
14/07/19 15:32:17 INFO mapred.JobClient: Virtual memory (bytes) snapshot=7732314112
14/07/19 15:32:17 INFO mapred.JobClient: Total committed heap usage (bytes)=2022703104
14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Transferred 8 bytes in 43.3403 seconds (0.1846 bytes/sec)
14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Retrieved 0 records.
14/07/19 15:32:17 ERROR tool.ImportTool: Error during import: Import job failed!
昨天發了從DB2中導入數據的錯誤日誌，後來定位爲JDBC驅動包的問題，但是替換之後還是沒用，然後又定位爲是由於JDBC線程未關閉，然後換成了從Oracle中導入數據，又來了另外一個問題：連接有時能通，有時不能通，對於-m 4是基本通不了，原因在於Mapreduce是自動分發匹配的，所以有些結點可能是不可見的，這樣對我們連接又會出現網絡不通的問題，初步解決方案：自行修改源代碼，找到任務分發代碼，直接指定一個專門用來做數據導入的結點。正在實踐ing。。。。