Unable to Start Clusters on GCP - Clusters Stuck in "CREATING" State

[etlundquist]

I set up my Databricks Account on GCP via GCP Marketplace and then created my first workspace via the Accounts Console (default Databricks VPC). Everything seemed to be ok until I attempted to create my first cluster. The cluster hangs indefinitely in CREATING and I don't see any compute instances / GKE clusters being spun up in the account. I can see the GCS storage buckets and service account Databricks created as part of workspace creation (with the required roles: Compute Storage Admin, K8s Engine Admin, Databricks Service IAM Role for Workspace), but no compute resources seem to be provisioned when I attempt to launch the cluster from the Workspace Console. I'm a project owner for the GCP project in which the workspace is launched, so it shouldn't be a permissions issue for my user account. I'm not sure exactly what to check as there's no meaningful logs in terms of why it's hanging or where the error lies. Any help on where to look or what to check would be greatly appreciated!

[Prabakar]

One possible reason for this issue could be that the Databricks service account does not have the necessary permissions to create resources in your GCP project.

To resolve this issue, you can try granting the Databricks service account the necessary permissions to create resources in your GCP project. You can follow the steps outlined in the Required permissions | Databricks on Google Cloud to grant the necessary permissions to the Databricks service account.

[etlundquist]

The databricks service account used to create the workspace is a project owner, so that shouldn't be the case. If there are additional permissions above and beyond what's granted with "Owner" status let me know, but I don't think that's the issue here. I do notice that the workspace-specific service accounts that get created have the format:

db-{workspace_id}@prod-gcp-us-central1.iam.gserviceaccount.com

where the project ID is `prod-gcp-us-central1` instead of the actual project specified when creating the workspace. Maybe that has something to do with it?

[Anonymous]

Hi @etlundquist 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!