We were able to resolve the above issue by increasing the quotas for two separate compute families. One was the Standard F4s family, which apparently the worker uses and the other was the Standard EDV4 family which I guess is the driver node. Now we'...
I am having a similar problem but in my case I get this error after about 20 minutes of the gateway cluster trying to start: com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 0829-204342-h9tqov5v...
