We can filter the data while importing
--where
bound val query will be framed based on the where clause CONDITIONS
[cloudera@quickstart ~]$ sqoop import --connect jdbc:mysql://quickstart.cloudera:3306/retail_db --username root --password cloudera --table orders --warehouse-dir /user/training/sqoop_import/retail_db --delete-target-dir --where "order_status IN('COMPLETE','CLOSED') AND order_date LIKE '2013-08%'"
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/11/02 03:55:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.0
19/11/02 03:55:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/11/02 03:55:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/11/02 03:55:24 INFO tool.CodeGenTool: Beginning code generation
19/11/02 03:55:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
19/11/02 03:55:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
19/11/02 03:55:25 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/43d56214c257d4ebfea6c9f9661e8d6d/orders.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/11/02 03:55:28 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/43d56214c257d4ebfea6c9f9661e8d6d/orders.jar
19/11/02 03:55:30 INFO tool.ImportTool: Destination directory /user/training/sqoop_import/retail_db/orders deleted.
19/11/02 03:55:30 WARN manager.MySQLManager: It looks like you are importing from mysql.
19/11/02 03:55:30 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
19/11/02 03:55:30 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
19/11/02 03:55:30 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
19/11/02 03:55:30 INFO mapreduce.ImportJobBase: Beginning import of orders
19/11/02 03:55:30 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
19/11/02 03:55:30 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/11/02 03:55:30 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/11/02 03:55:30 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/11/02 03:55:38 INFO db.DBInputFormat: Using read commited transaction isolation
19/11/02 03:55:38 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`order_id`), MAX(`order_id`) FROM `orders` WHERE ( order_status IN('COMPLETE','CLOSED') AND order_date LIKE '2013-08%' )
19/11/02 03:55:38 INFO db.IntegerSplitter: Split size: 16853; Num splits: 4 from: 1297 to: 68710
19/11/02 03:55:38 INFO mapreduce.JobSubmitter: number of splits:4
19/11/02 03:55:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1572629054486_0014
19/11/02 03:55:39 INFO impl.YarnClientImpl: Submitted application application_1572629054486_0014
19/11/02 03:55:39 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1572629054486_0014/
19/11/02 03:55:39 INFO mapreduce.Job: Running job: job_1572629054486_0014
19/11/02 03:55:51 INFO mapreduce.Job: Job job_1572629054486_0014 running in uber mode : false
19/11/02 03:55:51 INFO mapreduce.Job: map 0% reduce 0%
19/11/02 03:56:19 INFO mapreduce.Job: map 25% reduce 0%
19/11/02 03:56:21 INFO mapreduce.Job: map 50% reduce 0%
19/11/02 03:56:22 INFO mapreduce.Job: map 75% reduce 0%
19/11/02 03:56:23 INFO mapreduce.Job: map 100% reduce 0%
19/11/02 03:56:23 INFO mapreduce.Job: Job job_1572629054486_0014 completed successfully
19/11/02 03:56:23 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=687392
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=472
HDFS: Number of bytes written=102579
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Killed map tasks=1
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=107831
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=107831
Total vcore-milliseconds taken by all map tasks=107831
Total megabyte-milliseconds taken by all map tasks=110418944
Map-Reduce Framework
Map input records=2517
Map output records=2517
Input split bytes=472
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=2883
CPU time spent (ms)=12440
Physical memory (bytes) snapshot=706289664
Virtual memory (bytes) snapshot=6262525952
Total committed heap usage (bytes)=630718464
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=102579
19/11/02 03:56:23 INFO mapreduce.ImportJobBase: Transferred 100.1748 KB in 53.1503 seconds (1.8847 KB/sec)
19/11/02 03:56:23 INFO mapreduce.ImportJobBase: Retrieved 2517 records.
[cloudera@quickstart ~]$ hdfs dfs -ls /user/training/sqoop_import/retail_db/order_itemsFound 5 items
-rw-r--r-- 1 cloudera supergroup 0 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/_SUCCESS
-rw-r--r-- 1 cloudera supergroup 444360 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00000
-rw-r--r-- 1 cloudera supergroup 458847 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00001
-rw-r--r-- 1 cloudera supergroup 459277 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00002
-rw-r--r-- 1 cloudera supergroup 459339 2019-11-02 03:49 /user/training/sqoop_import/retail_db/order_items/part-m-00003
[cloudera@quickstart ~]$ hdfs dfs -ls /user/training/sqoop_import/retail_db/orderS
ls: `/user/training/sqoop_import/retail_db/orderS': No such file or directory
[cloudera@quickstart ~]$ hdfs dfs -ls /user/training/sqoop_import/retail_db/orders
Found 5 items
-rw-r--r-- 1 cloudera supergroup 0 2019-11-02 03:56 /user/training/sqoop_import/retail_db/orders/_SUCCESS
-rw-r--r-- 1 cloudera supergroup 85720 2019-11-02 03:56 /user/training/sqoop_import/retail_db/orders/part-m-00000
-rw-r--r-- 1 cloudera supergroup 0 2019-11-02 03:56 /user/training/sqoop_import/retail_db/orders/part-m-00001
-rw-r--r-- 1 cloudera supergroup 0 2019-11-02 03:56 /user/training/sqoop_import/retail_db/orders/part-m-00002
-rw-r--r-- 1 cloudera supergroup 16859 2019-11-02 03:56 /user/training/sqoop_import/retail_db/orders/part-m-00003
[cloudera@quickstart ~]$
No comments:
Post a Comment