Automatic Identity Management

andrefilipemm — Mon, 26 Jan 2026 17:10:24 GMT

Hello,

In the context of reviewing our company's databricks structure and migrating legacy workspaces to Unity Catalog enabled ones, we're stuck with a few questions regarding enabling the automatic identity management feature.

We currently provision Databricks users and groups using Terraform, pulling group membership data from Microsoft Entra ID. Our approach involves:

Manual Replication: For each Databricks workspace, we manually replicate Entra ID group definitions in Terraform.
Terraform Resources: We use the Databricks Terraform provider to create:
- databricks_group resources for each group
- databricks_user resources for individual users
- databricks_service_principal resources for service principals
- databricks_group_member resources to manage memberships
Limitations: This approach requires manual updates and Terraform deployments for every group or membership change.

Questions

If we enable the AIM will there be user/groups collision? My understand says otherwise, considering that we currently provision local workspace user/groups and AIM works at the databricks account level. Hence, the groups that'll be generated via AIM will be account sourced and shouldn't conflict with workspace provisioned ones.
Suppose there is no group collision. The next step to take after enabling AIM is simply destroying all workspace local users/groups since these will no longer be needed?
A follow-up question to 2: what happens to workspace resources which are bound to workspace users and groups?

Re: Automatic Identity Management

SteveOstrowski — Sun, 08 Mar 2026 07:21:02 GMT

Hi @andrefilipemm,

These are important questions when planning a migration to identity federation and Unity Catalog. Let me address each one.

BACKGROUND: HOW IDENTITY FEDERATION WORKS

When you enable identity federation on a workspace, identity management shifts from the workspace level to the account level. Users and service principals that were created at the workspace level are automatically recognized at the account level (they share the same underlying identity based on email/applicationId). Groups, however, behave differently: workspace-local groups remain workspace-scoped and do NOT automatically become account groups.

Databricks documentation on identity best practices:
https://docs.databricks.com/aws/en/admin/users-groups/best-practices

QUESTION 1: WILL ENABLING IDENTITY FEDERATION CAUSE USER/GROUP COLLISIONS?

For users and service principals: No collisions. Workspace-level users and service principals are already synced to the account level based on their email address or application ID. When you enable identity federation, those identities are recognized as the same identity at both levels. There is no duplication.

For groups: This is where you need to be careful. Workspace-local groups and account groups are separate objects, even if they share the same name. When you enable identity federation, your existing workspace-local groups remain as workspace-local groups. They do not automatically merge with or collide with account-level groups. However, if you then create an account group with the same name and assign it to the workspace, you will have two groups with identical names (one workspace-local, one account-level), which can cause confusion.

The recommended approach is to rename your workspace-local groups (e.g., append " (workspace)" to the name) before creating equivalent account groups. This avoids any ambiguity.

If you are currently using workspace-level SCIM provisioning via Terraform, you should be aware that workspace-level SCIM does not recognize account groups and API calls will fail if they involve account groups. You will need to transition your SCIM provisioning to the account level.

QUESTION 2: SHOULD WORKSPACE-LOCAL USERS AND GROUPS BE DESTROYED AFTER ENABLING IDENTITY FEDERATION?

Users and service principals: You do NOT need to destroy workspace-level users and service principals. They are the same identity at both levels. Removing them at the workspace level simply removes their workspace assignment, it does not delete the account-level identity.

Groups: Yes, after migration you should eventually remove workspace-local groups, but only AFTER you have:

1. Created equivalent account-level groups with the same memberships
2. Migrated all permissions from the workspace-local groups to the new account groups
3. Verified that everything works correctly with the account groups

The recommended four-step process from the Databricks documentation is:

1. Migrate workspace-level SCIM provisioning to account-level SCIM (set up your Entra ID SCIM connector at the account level and disable the workspace-level one)
2. Rename workspace-local groups (e.g., append " (workspace)") to avoid name conflicts
3. Create account groups, assign them to the workspace, grant equivalent permissions using the Permissions API or the UCX migration utilities
4. Delete the original workspace-local groups once you have confirmed the account groups are working

Documentation on converting workspace-local groups:
https://docs.databricks.com/aws/en/admin/users-groups/workspace-local-groups

QUESTION 3: WHAT HAPPENS TO RESOURCES BOUND TO WORKSPACE-LOCAL USERS AND GROUPS?

Permissions on workspace-level objects (notebooks, folders, jobs, clusters, etc.) that were granted to workspace-local users and service principals will continue to work because those identities are the same at both the workspace and account levels.

Permissions granted to workspace-local groups, however, will need to be migrated. Workspace-local groups cannot be used with Unity Catalog, cannot be assigned to other workspaces, and cannot be granted account-level roles. When you delete a workspace-local group, any permissions tied to it are lost.

Before deleting workspace-local groups, you must explicitly migrate their permissions to the replacement account groups. You can do this using:

- The Databricks Permissions API to read and re-apply workspace object permissions
- The UCX (Unity Catalog migration) utilities, which include tooling for group permission migration

This is the most critical step in the migration, so take your time and validate thoroughly before removing the old groups.

TERRAFORM-SPECIFIC GUIDANCE

Since you are currently using Terraform with the Databricks provider, here is the transition path:

1. Set up account-level SCIM provisioning from Entra ID. You can do this through an Azure Enterprise Application configured as an account-level SCIM connector. Documentation:
https://docs.databricks.com/aws/en/admin/users-groups/scim/aad

2. In your Terraform code, shift from workspace-level resources (databricks_group, databricks_user at the workspace provider) to account-level resources using the account-level Databricks provider. Use databricks_mws_permission_assignment to assign account-level groups to workspaces.

3. Use the databricks_group data source at the account level to reference groups synced from Entra ID rather than manually replicating them.

This eliminates the manual group replication you described and centralizes identity management at the account level, which is the intended architecture for Unity Catalog environments.

SUMMARY

- Enabling identity federation does not cause collisions for users/service principals. Groups need careful handling (rename, recreate, migrate permissions, delete old ones).
- Do not simply destroy workspace-local groups. Follow the four-step migration process to preserve permissions.
- Resource permissions tied to users/service principals carry over. Permissions tied to workspace-local groups must be explicitly migrated to account groups before deletion.

Hope this helps with your migration planning.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

topic Re: Automatic Identity Management in Administration & Architecture

Automatic Identity Management

Re: Automatic Identity Management