How does spark-submit.sh work with different modes and different cluster managers?
20:48 13 Mar 2020

In Apache Spark, how does spark-submit.sh work with different modes and different cluster managers? Specifically:

In local deployment mode,

  • does spark-submit.sh skip calling any cluster manager?
  • Is it correct that there is no need to install a cluster manager on the local machine?

In client or cluster deployment mode,

  • Does spark-submit.sh work with different cluster managers (Spark standalone, YARN, Mesos, Kubernetes)? Do different cluster managers have different interfaces, and spark-submit.sh has to invoke them in different ways?

  • Does spark-submit.sh appear to programmers the same interface except --master? option --master of spark-submit.sh is used to specify a cluster manager.

apache-spark cluster-computing