Powered By Blogger

Sunday, February 9, 2020

Loading local jar in spark-shell

Loading jar from local
-bash-4.2$ spark-shell --jars /home_dir/basan/dpp/pipelinecore_2.11-1.2.1.jar


-bash-4.2$ spark-shell --jars


Loading jar from internet for spark shell


-bash-4.2$ spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2 

Loading jar from internet for spark submit


spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2 --queue basanqueue --driver-memory 2G --num-executors 8 --executor-memory 14G --executor-cores 6 --class com.basan.CoreTableProcessor  dpp-assembly-1.0.jar application.conf 1 2

No comments:

Post a Comment