topic Re: [GCP] Failed to migrate a project onto an organization in Data Engineering

[GCP] Failed to migrate a project onto an organization

AlainT — Wed, 17 Jul 2024 15:46:09 GMT

Hi,

After migrating a project to an organization, we are unable to create a workspace without encountering errors. Previously working workspaces are also failing.

I have granted admin/owner access to all users who need Databricks. The latest error involves a mishmash with the service account, leading to its auto-deletion, and ultimately affecting GKE clusters.

I tried creating a custom VPC following the documentation (https://docs.gcp.databricks.com/en/security/network/classic/customer-managed-vpc.html#overview), but encountered a 400 error on step 3.7.d.

Additionally, I have configured `constraints/iam.allowedPolicyMemberDomains` with information from the Domain Restricted Sharing documentation here: https://docs.gcp.databricks.com/en/admin/account-settings-gcp/create-subscription.html#create-a-subscription

I found my Organization ID using this command: gcloud organizations describe YOUR_ORGANIZATION_ID

However, I'm interested in knowing how to confirm this configuration, as it currently triggers alerts when changing user permissions in my project where Databricks is used:

"IAM policy update failed
The 'Domain Restricted Sharing' organization policy (constraints/iam.allowedPolicyMemberDomains) is enforced. Only principals in allowed domains can be added as principals in the policy. Correct the principal emails and try again. Learn more about domain restricted sharing."

Could errors on creations of workspace to be related to a new billing plan configuration that needs to be done?

Two days of debugging has been enough 😅

Any assistance or suggestions would be greatly appreciated!

Re: [GCP] Failed to migrate a project onto an organization

AlainT — Thu, 18 Jul 2024 13:20:04 GMT

Hi @Retired_mod

I'm still checking all access and all IAM policies. Because, my question still which are "all necessary domains" and what's "all necessary IAM roles and permissions are correctly assigned" and how to test it.

Note that I don't create a service account my self, but it's done by Databricks environment automatically.

Because I don't know the process behind the automatic creation of the workspace and creation of resources on GCP, I cannot be sure of where is the problem. I must use deduction and read docs, a lot of docs.

Maybe, it could be useful for future debugging to create some CLI for testing configuration on GCP.

Another solution for me, is to create all from scratch by myself on GCP instead of the automatic solution and configure a custom environment on Databricks, but it's my first experience on Databricks 😉

Voilà 🙄

Re: [GCP] Failed to migrate a project onto an organization

AlainT — Thu, 18 Jul 2024 14:28:07 GMT

The solution is...

I had finaly edited constraints/iam.allowedPolicyMemberDomains on project and on organization with C01p0oudw (Databricks customer_id on GCP) and our customer_id (gcloud organizations describe YOUR_ORGANIZATION_ID). Not only on organization, or maybe I had wrong encoded it previously on project. (It's was well encoded on organization but, i think, badly on project).

Databricks doc here (point 1, second dot) : https://docs.gcp.databricks.com/en/admin/account-settings-gcp/create-subscription.html#create-a-subscription

Thx for your assistance @Retired_mod 😉