Hi everyone,

I'm encountering an intermittent issue when launching a Databricks pipeline cluster. Error message

com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster xxxx-xxxxxx-ofgxxxxx: Attempt to launch cluster with invalid arguments. databricks_error_message: Spark failed to start: Driver unresponsive. Possible reasons: library conflicts, incorrect metastore configuration, and i... This error is likely due to a misconfiguration in the pipeline. Check the pipeline cluster configuration and associated cluster policy.

Interestingly, the pipeline fails 3 times with this error and then succeeds on the 4th attempt without any manual intervention.
Has anyone experienced something similar? What could be causing this  "Spark failed to start: Driver unresponsive" error? Are there known configurations or best practices i should check to present this from happening in future?

Any insights would be greatly appreciated.
Thanks in advance!