topic Re: Databricks shared workspace in Administration & Architecture

Databricks shared workspace

nskiran1 — Thu, 06 Feb 2025 13:02:02 GMT

We have a Self service portal through which users can launch databricks clusters of different configurations.

This portal is set up to work in Dev, Sandbox and Prod environments. We have configured databricks workspaces only for Sandbox and Prod portals only. So, users can launch databricks clusters through Sandbox and Prod portals. No databricks workspace available for Dev Portal.

Self service portal invokes different databricks APIs like list/delete/register instance profiles, create/delete clusters etc using Python. We have set up Service Principals for all the databricks workspaces and invoke databricks APIs with respective service principals.

Recently, our management decided to route databricks cluster launch requests from Dev portal to Sandbox databricks workspace on urgent basis as we do not have databricks workspace for Dev portal

Is it possible to have databricks 'shared' workspace that can be tied to multiple AWS accounts? Can someone share documentation on IAM permissions policies on how to configure multiple AWS accounts for shared workspace please?

Re: Databricks shared workspace

Alberto_Umana — Thu, 06 Feb 2025 13:03:57 GMT

Hello @nskiran1,

Yes, it is possible to have a 'shared' workspace that can be tied to multiple AWS accounts. This can be achieved by associating multiple VPCs (Virtual Private Clouds) across different AWS accounts with a single Databricks account

You should associate your VPCs and create a cross account IAM role

https://docs.databricks.com/en/admin/account-settings-e2/credentials.html#step-1-create-a-cross-account-iam-role

https://docs.databricks.com/en/security/network/classic/customer-managed-vpc.html

https://docs.databricks.com/en/admin/account-settings-e2/credentials.html

Re: Databricks shared workspace

nskiran — Mon, 10 Feb 2025 07:34:58 GMT

@Alberto_Umana Thanks for sharing doc links

We have exact same set up to support shared databricks workspace. But still Im facing issue while adding instance profile

I am trying to add AWS Instance Profile created in source AWS Account (No databricks workspace) to a target AWS Account to which databricks workspace set up available. Is this possible?

I have added required IAM permissions for both instance profile as well as cross account role. What else am I missing here?

{"error_code":"DRY_RUN_FAILED","message":"Verification of the instance profile failed. AWS error: You are not authorized to perform this operation.","details":[{"@type":"type.googleapis.com/google.rpc.ErrorInfo","reason":"CM_API_ERROR_SOURCE_CALLER_ERROR","domain":""}]}