Databricks Community

smukherjee · 4 weeks ago

How to migrate an Azure Databricks to a different Azure tenant?

Does DAB is the best method if a databricks workspace is freshly built on top of just migrating the the workspace resources only like notebook, jobs, cluster, policies, permissions etc?
Please share detailed steps to achieve this.

stephen4 · 3 weeks ago

A fresh workspace makes DAB a reasonable option, especially since you mainly need to migrate notebooks, jobs, policies, clusters, and permissions.

Martínezluas · 3 weeks ago

Yes, DAB is a good approach if the new workspace is already built and you mainly need to migrate notebooks, jobs, pipelines, and other workspace resources. You can use databricks bundle generate to create configuration for existing resources, put it under version control, then deploy the bundle to the new workspace; just keep in mind that identities, secrets, Unity Catalog objects, and other external dependencies may need to be handled separately.

anuj_lathi · Friday

Short answer: Yes, bundles are a good fit for the workspace-resource layer (jobs, pipelines, clusters, policies, dashboards, warehouses, permissions). They won't move everything, though. In practice, a cross-tenant move is a rebuild of the workspace with bundles replaying your code and config on top, plus separate work for identity, data, and infrastructure.

What bundles cover and what they don't

Bundles are an IaC approach: YAML plus source files, deployed with the Databricks CLI [1]. Supported resource types include jobs, pipelines, clusters, cluster policies, instance pools, SQL warehouses, dashboards, alerts, experiments, secret scopes, model serving endpoints, and Unity Catalog objects such as catalogs, schemas, volumes and registered models [2]. Resources can also carry permissions blocks [2].

Bundles do not handle these, so plan them separately:

Azure infrastructure: the workspace itself, VNet injection/Private Link, storage accounts, access connectors, Key Vault. Use Terraform/Bicep/ARM in the new tenant.
Identity: the new tenant has a different Entra ID, so users, groups and service principals have new identities. Provision them in the new account first (SCIM or Entra ID provisioning).
Data: Delta tables and files in ADLS don't move with a bundle. Copy them at the storage level, or use deep clone or Delta Sharing, depending on volume.
Secret values: a bundle can define a scope but not carry the secret values over. Re-create them.

Steps

1. Stand up the target. Create the new workspace and Unity Catalog metastore/storage credentials/external locations in the new tenant with IaC. Re-create users, groups and service principals, and assign them to the workspace.

2. Configure the CLI for both workspaces. Use OAuth U2M profiles, one for source and one for target [1]:

databricks auth login --host https://adb-<old>.azuredatabricks.net --profile old
databricks auth login --host https://adb-<new>.azuredatabricks.net --profile new
databricks --version   # bundles need v0.218.0 or above

3. Generate bundle config from the source workspace. You don't need to hand-write YAML for existing resources, because databricks bundle generate produces it [2]:

databricks bundle init   # or start from an empty databricks.yml
databricks bundle generate job --existing-job-id <job-id> --profile old

It also pulls down the notebooks the job references. Repeat for pipelines, dashboards and so on. For anything that has no generate support (for example, cluster policies), write the YAML yourself, using the resource reference as the schema [2].

4. Get notebooks into source control. Ideally they already live in Git. If some only exist in the workspace, export them with databricks workspace export-dir and commit them, so the repo becomes the source of truth.

5. Add a target for the new tenant and remap identities. Permissions in the YAML must reference principals that exist in the new tenant. User emails often change, so parameterize them:

bundle:
  name: workspace-migration

variables:
  admin_group:
    default: data-platform-admins

targets:
  new_tenant:
    mode: production
    workspace:
      host: https://adb-<new>.azuredatabricks.net

resources:
  jobs:
    nightly_etl:
      name: nightly_etl
      permissions:
        - group_name: ${var.admin_group}
          level: CAN_MANAGE
      # tasks, clusters, schedule ...

Also update anything hard-coded to the old environment, such as storage paths, catalog names, warehouse IDs, cluster policy IDs, and secret scope references.

6. Validate and deploy.

databricks bundle validate -t new_tenant --profile new
databricks bundle deploy   -t new_tenant --profile new

validate warns about unknown resource properties [2], which catches typos early.

7. Re-apply permissions that aren't tied to a bundle resource. Workspace object ACLs are managed per object type (CAN MANAGE, CAN RUN, CAN ATTACH TO, and so on) [3]. For notebooks and folders that aren't bundle resources, apply permissions at the folder level so objects inherit them [3]. Unity Catalog grants also need re-applying in the new metastore, which is best scripted as SQL or Terraform.

8. Migrate data, then test. Copy the data, repoint tables, run jobs in the new workspace, and compare outputs against the source before cutting over schedules.

Recommendation

Use Terraform/Bicep for the Azure infrastructure, workspace and account-level identity.
Use bundles for jobs, pipelines, compute definitions, dashboards, and their permissions.
Use Git for notebooks.
Use a separate data copy process for tables and files.

That split gives you a repeatable, reviewable migration, and the same bundle then doubles as your CI/CD deployment going forward.