Databricks Community

All applications stuck in Waiting State on Standalone Spark Cluster

Spark Standalone Cluster Configuration (Spark 3.0.0)- 1 Master2 Workers (4 cores each) I am using Airflow SparkSubmitOperator to submit the job to Spark Master in Cluster mode. There are multiple(~20) DAGs on airflow submitting jobs to Spark. These ...