Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
If I understand correctly, Spark driver is a master process. Is it the same as the Spark Master. I get confused with the Spark master and Spark driver.
This is a common misconception. Spark Master and Spark driver are two independent and isolated JVM's running on the same instance. Spark Master's responsibilities are to ensure the Spark worker's daemons are up and running and monitor the health. Also, manage the resource on the Spark cluster. Spark Master acts as the cluster manager in the Spark Standalone architecture. The Spark driver is the brain of the Spark engine.
This is a common misconception. Spark Master and Spark driver are two independent and isolated JVM's running on the same instance. Spark Master's responsibilities are to ensure the Spark worker's daemons are up and running and monitor the health. Also, manage the resource on the Spark cluster. Spark Master acts as the cluster manager in the Spark Standalone architecture. The Spark driver is the brain of the Spark engine.
I have a question regarding your answer: - what is the role of Master and Driver if i spark-submit from different machine. Specifically does this roles or responsibilities change if i use cluster-mode or client-mode?
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
