Powered By Blogger

Saturday, November 2, 2019

sqoop boundary query

[cloudera@quickstart ~]$ sqoop import --connect jdbc:mysql://quickstart.cloudera:3306/retail_db --username root --password cloudera --table order_items --warehouse-dir /user/training/sqoop_import/retail_db --delete-target-dir --boundary-query "SELECT min(order_item_order_id),max(order_item_order_id) FROM order_items WHERE order_item_order_id>=10000"
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/11/02 03:48:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.0
19/11/02 03:48:59 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/11/02 03:49:00 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/11/02 03:49:00 INFO tool.CodeGenTool: Beginning code generation
19/11/02 03:49:00 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `order_items` AS t LIMIT 1
19/11/02 03:49:00 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `order_items` AS t LIMIT 1
19/11/02 03:49:00 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/bab6092b5021631680fbfc4a3ceda418/order_items.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/11/02 03:49:04 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/bab6092b5021631680fbfc4a3ceda418/order_items.jar
19/11/02 03:49:05 INFO tool.ImportTool: Destination directory /user/training/sqoop_import/retail_db/order_items deleted.
19/11/02 03:49:05 WARN manager.MySQLManager: It looks like you are importing from mysql.
19/11/02 03:49:05 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
19/11/02 03:49:05 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
19/11/02 03:49:05 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
19/11/02 03:49:05 INFO mapreduce.ImportJobBase: Beginning import of order_items
19/11/02 03:49:05 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
19/11/02 03:49:05 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/11/02 03:49:05 WARN db.DataDrivenDBInputFormat: Could not find $CONDITIONS token in query: SELECT min(order_item_order_id),max(order_item_order_id) FROM order_items WHERE order_item_order_id>=10000; splits may not partition data.
19/11/02 03:49:05 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/11/02 03:49:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/11/02 03:49:14 INFO db.DBInputFormat: Using read commited transaction isolation
19/11/02 03:49:14 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT min(order_item_order_id),max(order_item_order_id) FROM order_items WHERE order_item_order_id>=10000
19/11/02 03:49:14 INFO db.IntegerSplitter: Split size: 14720; Num splits: 4 from: 10000 to: 68883
19/11/02 03:49:15 INFO mapreduce.JobSubmitter: number of splits:4
19/11/02 03:49:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1572629054486_0013
19/11/02 03:49:16 INFO impl.YarnClientImpl: Submitted application application_1572629054486_0013
19/11/02 03:49:16 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1572629054486_0013/
19/11/02 03:49:16 INFO mapreduce.Job: Running job: job_1572629054486_0013
19/11/02 03:49:26 INFO mapreduce.Job: Job job_1572629054486_0013 running in uber mode : false
19/11/02 03:49:26 INFO mapreduce.Job:  map 0% reduce 0%
19/11/02 03:49:52 INFO mapreduce.Job:  map 25% reduce 0%
19/11/02 03:49:54 INFO mapreduce.Job:  map 50% reduce 0%
19/11/02 03:49:57 INFO mapreduce.Job:  map 100% reduce 0%
19/11/02 03:49:58 INFO mapreduce.Job: Job job_1572629054486_0013 completed successfully
19/11/02 03:49:58 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=688156
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=513
HDFS: Number of bytes written=1821823
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Killed map tasks=1
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=100430
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=100430
Total vcore-milliseconds taken by all map tasks=100430
Total megabyte-milliseconds taken by all map tasks=102840320
Map-Reduce Framework
Map input records=58884
Map output records=58884
Input split bytes=513
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=2270
CPU time spent (ms)=22030
Physical memory (bytes) snapshot=817684480
Virtual memory (bytes) snapshot=6256640000
Total committed heap usage (bytes)=668467200
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=1821823
19/11/02 03:49:58 INFO mapreduce.ImportJobBase: Transferred 1.7374 MB in 52.6162 seconds (33.8132 KB/sec)
19/11/02 03:49:58 INFO mapreduce.ImportJobBase: Retrieved 58884 records.
[cloudera@quickstart ~]$ hdfs dfs -ls /user/training/sqoop_import/retail_db/order_itemsFound 5 items
-rw-r--r--   1 cloudera supergroup          0 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/_SUCCESS
-rw-r--r--   1 cloudera supergroup     444360 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00000
-rw-r--r--   1 cloudera supergroup     458847 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00001
-rw-r--r--   1 cloudera supergroup     459277 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00002
-rw-r--r--   1 cloudera supergroup     459339 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00003
[cloudera@quickstart ~]$ 

No comments:

Post a Comment