1、查詢hive表中的student數據及信息
hive> select * from student;
OK
001 xiaohong 002
002 xiaolan 002
001 xiaohong 003
002 xiaolan 003
Time taken: 0.114 seconds, Fetched: 4 row(s)
hive> desc formatted student;
OK
# col_name data_type comment
id string
name string
# Partition Information
# col_name data_type comment
classroom string
# Detailed Table Information
Database: default
Owner: root
CreateTime: Thu Mar 19 14:00:19 CST 2020
LastAccessTime: UNKNOWN
Retention: 0
Location: hdfs://master:9000/user/hive/warehouse/student
Table Type: MANAGED_TABLE
Table Parameters:
transient_lastDdlTime 1584597619
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
field.delim ,
serialization.format ,
Time taken: 0.065 seconds, Fetched: 32 row(s)
從上面的信息可知，student是以classroom爲分區進行存儲的；數據存儲位置爲/user/hive/warehouse/student；
2、查看hive中student表數據的存儲位置
[root@master ~]# hdfs dfs -ls /user/hive/warehouse/student/
Found 2 items
drwxr-xr-x - root supergroup 0 2020-03-19 14:04 /user/hive/warehouse/student/classroom=002
drwxr-xr-x - root supergroup 0 2020-03-19 14:10 /user/hive/warehouse/student/classroom=003
[root@master ~]# hdfs dfs -ls /user/hive/warehouse/student/classroom=002
Found 1 items
-rwxr-xr-x 1 root supergroup 25 2020-03-19 14:04 /user/hive/warehouse/student/classroom=002/stu.txt
[root@master ~]# hdfs dfs -cat /user/hive/warehouse/student/classroom=002/stu.txt
001,xiaohong
002,xiaolan
由此可知根據逗號分隔數據；
3、先在mysql中創建字段和類型相同的表
mysql> create table student(id varchar(255),name varchar(255));
Query OK, 0 rows affected (0.02 sec)
mysql> select * from student;
Empty set (0.00 sec)
mysql>
4、sqoop導入數據
[root@master bin]# ./sqoop export --connect "jdbc:mysql://192.168.230.21:3306/mysql" --username root --password 123456 --table student --fields-terminated-by "," --export-dir "/user/hive/warehouse/student/classroom=002";
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/06/29 17:27:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
20/06/29 17:27:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/06/29 17:27:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
20/06/29 17:27:58 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/softWare/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hbase/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hive/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/06/29 17:27:58 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:27:58 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:27:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/softWare/hadoop/hadoop-2.7.3
Note: /tmp/sqoop-root/compile/ee585e722d7b5a438d67d53c369ed36e/student.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
20/06/29 17:28:00 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/ee585e722d7b5a438d67d53c369ed36e/student.jar
20/06/29 17:28:00 INFO mapreduce.ExportJobBase: Beginning export of student
20/06/29 17:28:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
20/06/29 17:28:01 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
20/06/29 17:28:01 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:28:01 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/06/29 17:28:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
20/06/29 17:28:06 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:28:06 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:28:06 INFO mapreduce.JobSubmitter: number of splits:4
20/06/29 17:28:06 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:28:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1593394152340_0027
20/06/29 17:28:07 INFO impl.YarnClientImpl: Submitted application application_1593394152340_0027
20/06/29 17:28:07 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1593394152340_0027/
20/06/29 17:28:07 INFO mapreduce.Job: Running job: job_1593394152340_0027
20/06/29 17:28:16 INFO mapreduce.Job: Job job_1593394152340_0027 running in uber mode : false
20/06/29 17:28:16 INFO mapreduce.Job: map 0% reduce 0%
20/06/29 17:28:37 INFO mapreduce.Job: map 50% reduce 0%
20/06/29 17:28:38 INFO mapreduce.Job: map 100% reduce 0%
20/06/29 17:28:38 INFO mapreduce.Job: Job job_1593394152340_0027 completed successfully
20/06/29 17:28:38 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=551488
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=764
HDFS: Number of bytes written=0
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=4
Data-local map tasks=4
Total time spent by all maps in occupied slots (ms)=71778
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=71778
Total vcore-milliseconds taken by all map tasks=71778
Total megabyte-milliseconds taken by all map tasks=73500672
Map-Reduce Framework
Map input records=2
Map output records=2
Input split bytes=681
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=799
CPU time spent (ms)=3130
Physical memory (bytes) snapshot=445382656
Virtual memory (bytes) snapshot=8325513216
Total committed heap usage (bytes)=170655744
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
20/06/29 17:28:38 INFO mapreduce.ExportJobBase: Transferred 764 bytes in 36.6371 seconds (20.8532 bytes/sec)
20/06/29 17:28:38 INFO mapreduce.ExportJobBase: Exported 2 records.
[root@master bin]# ./sqoop export --connect "jdbc:mysql://192.168.230.21:3306/mysql" --username root --password 123456 --table student --fields-terminated-by "," --export-dir "/user/hive/warehouse/student/classroom=003";
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/06/29 17:29:21 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
20/06/29 17:29:21 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/06/29 17:29:22 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
20/06/29 17:29:22 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/softWare/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hbase/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hive/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/06/29 17:29:22 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:29:22 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:29:22 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/softWare/hadoop/hadoop-2.7.3
Note: /tmp/sqoop-root/compile/b4bafb65b47b2cded3caa3989b6e4d00/student.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
20/06/29 17:29:24 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/b4bafb65b47b2cded3caa3989b6e4d00/student.jar
20/06/29 17:29:24 INFO mapreduce.ExportJobBase: Beginning export of student
20/06/29 17:29:24 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
20/06/29 17:29:25 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
20/06/29 17:29:25 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:29:25 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/06/29 17:29:25 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
20/06/29 17:29:31 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:29:31 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:29:31 INFO mapreduce.JobSubmitter: number of splits:4
20/06/29 17:29:31 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:29:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1593394152340_0028
20/06/29 17:29:32 INFO impl.YarnClientImpl: Submitted application application_1593394152340_0028
20/06/29 17:29:32 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1593394152340_0028/
20/06/29 17:29:32 INFO mapreduce.Job: Running job: job_1593394152340_0028
20/06/29 17:29:41 INFO mapreduce.Job: Job job_1593394152340_0028 running in uber mode : false
20/06/29 17:29:41 INFO mapreduce.Job: map 0% reduce 0%
20/06/29 17:29:52 INFO mapreduce.Job: map 50% reduce 0%
20/06/29 17:29:54 INFO mapreduce.Job: map 75% reduce 0%
20/06/29 17:29:55 INFO mapreduce.Job: map 100% reduce 0%
20/06/29 17:29:55 INFO mapreduce.Job: Job job_1593394152340_0028 completed successfully
20/06/29 17:29:56 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=551488
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=764
HDFS: Number of bytes written=0
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Launched map tasks=4
Data-local map tasks=2
Rack-local map tasks=2
Total time spent by all maps in occupied slots (ms)=35902
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=35902
Total vcore-milliseconds taken by all map tasks=35902
Total megabyte-milliseconds taken by all map tasks=36763648
Map-Reduce Framework
Map input records=2
Map output records=2
Input split bytes=681
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=500
CPU time spent (ms)=2940
Physical memory (bytes) snapshot=448909312
Virtual memory (bytes) snapshot=8319729664
Total committed heap usage (bytes)=166592512
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
20/06/29 17:29:56 INFO mapreduce.ExportJobBase: Transferred 764 bytes in 31.064 seconds (24.5944 bytes/sec)
20/06/29 17:29:56 INFO mapreduce.ExportJobBase: Exported 2 records.
5、查看mysql中student表數據結果
mysql> select * from student;
+------+----------+
| id | name |
+------+----------+
| 001 | xiaohong |
| 002 | xiaolan |
| 001 | xiaohong |
| 002 | xiaolan |
+------+----------+
4 rows in set (0.00 sec)
mysql>