Passing function to spark
https://spark.apache.org/docs/latest/rdd-programming-guide.html#passing-functions-to-spark
Spark’s API relies on heavily passing function in the driver program to run on the cluster. There are three ways to pass function in driver program.
Anonymous function syntax- programmers heavily use lambda functions (anonymous function).
Static method in a global singleton object - defining a function using def first and then using it in driver program.
Passing by reference--it is being used to avoid sending the entire object, consider coping the fucition to a local variable.
https://spark.apache.org/docs/latest/rdd-programming-guide.html#passing-functions-to-spark
Spark’s API relies on heavily passing function in the driver program to run on the cluster. There are three ways to pass function in driver program.
Anonymous function syntax- programmers heavily use lambda functions (anonymous function).
Static method in a global singleton object - defining a function using def first and then using it in driver program.
Passing by reference--it is being used to avoid sending the entire object, consider coping the fucition to a local variable.
No comments:
Post a Comment