Databricks

avrm91 · ‎03-26-2024

I face the issue that my fresh new Databricks workspace is not capable to start any cluster.
"Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you."
After 1830 seconds (30,5 minutes) the warning is changing to:
"Failing to start. Clusters are failing to launch. Cluster launch will be retried. Details for the latest failure: Error 1 out of 2 pods scheduled. Failed to launch cluster in Kubernetes in 1830 seconds. Driver container readiness: false, last executors readiness ratio: 1/1, expected executors readiness ratio: 0.5. FailedToScaleUp caused by GCE quota exceeded: Node scale up in zones europe-west3-b associated with this pod failed: GCE quota exceeded. Pod is at risk of not being scheduled. Type: CLIENT_ERROR Code: GCP_QUOTA_EXCEEDED Cluster-id (internal): xxxx-xxxxxx-xxxxxxxx.
I tried another area europe-west1 instead of europe-west3 but I get the same error.
#GCP

Kaniz · ‎03-26-2024

Hi @avrm91,

Verify that your project has sufficient CPU quota in the Google Cloud Platform (GCP) project associated with your Databricks workspace. If the quota is exceeded, it can prevent cluster nodes from launching.
You can check your GCP quotas in the GCP Console under “IAM and Admin” > “Quotas.” Look for the relevant quota related to CPU or instances.
The library installer in Databricks is configured to time out after 3 minutes while fetching and installing jars. Network problems can cause this timeout.
To mitigate this issue, consider downloading the required libraries from Maven to a Databricks File ...².
If a cluster launches without all nodes, Azure Databricks automatically tries to acquire additional nodes and updates the cluster once available.
Consider configuring a cluster with a larger instance type and a smaller number of nodes to avoid th...³.