Spark Standalone Cluster Configuration (Spark 3.0.0)-
- 1 Master
- 2 Workers (4 cores each)
I am using Airflow SparkSubmitOperator to submit the job to Spark Master in Cluster mode. There are multiple(~20) DAGs on airflow submitting jobs to Spark. These DAGs are scheduled based on cron expressions. Also, all Spark jobs have the same configurations - 1 Driver core, 1 Executor with 1 core.
Now at some point Airflow is triggering 8 jobs simultaneously and for these 8 jobs corresponding 8 drivers are spawned on the Spark Workers. But as I have only 8 cores on my workers, these drivers end up taking all the cores and don't allow any executor to spawn, resulting in all 8 applications being in the WAITING state. Spark Master UI Snapshot. As you can see in the image the cluster is in kind of a deadlock state - nothing moves. The airflow keeps on submitting more applications which queues up - when I kill any of the WAITING state applications, a driver from the queued app spawns.
I can't change the cron expressions on DAGs to spread these applications out, as these are configured by the users. I tried to solve this problem by following approaches -
- Limit number of concurrent applications that can be run on the spark cluster - I couldn't find any such config for the Standalone Spark cluster
- Associate a dedicated worker for drivers - This way I can use one worker to spawn drivers and the other to spawn executors dedicatedly (Doesn't make much sense for a distributed system as if this node goes down the whole cluster is unusable - but in the current situation, I would take that). After lots of researching only suggestion, I could find was this - https://stackoverflow.com/a/45962256, But this one didn't work as suggested - the driver was still spawning on both the workers.
Is there any other approach or configuration that I can try to resolve this issue?
