DABs Migration Guide: Terraform to the Direct Deployment Engine

[himanshu_ranjan]

This guide covers the migration procedure and several behaviors worth knowing that this migration surfaced in practice.

Background and timeline

The deployment backend for DABs is moving from the Terraform engine to the new direct deployment engine. The direct engine talks to the Databricks REST APIs directly. It no longer uses the Terraform provider or a terraform.tfstate file, which removes a common source of state-related deployment failures and makes deployments simpler and faster.

The transition is already in progress:

Date	Change
Jun 10, 2026 (CLI 1.3.0)	Direct engine becomes the default for new bundles. Existing Terraform bundles keep their engine until migrated.
Jul 24, 2026 - Aug 11, 2026	Direct engine becomes the default for all bundles deployed in DABs-in-the-Workspace. Bundles still on Terraform are auto-migrated.
Aug 26, 2026	Terraform deployments will be auto-migrated for CLI v 1.14.0 or later.
Sep 2026	Terraform engine is disabled in new CLI releases.
Later	The Terraform engine is removed entirely.

Sources: What's coming and the Bundles release notes. The official guidance states that "the Terraform deployment engine will soon be deprecated."

Why migrate proactively: the migration will happen either way. Doing it on your own schedule lets you test each step and fix any issues in advance, rather than being migrated automatically on the cutover date.

Standard migration procedure

For a bundle currently on the Terraform engine, migrate one target at a time (dev → staging → prod), starting with the lowest-risk environment.

# 1. Deploy on Terraform first so the state is current (recommended).
databricks bundle deploy --target <env>

# 2. Add `engine: direct` to databricks.yml, then convert the state.
#    This reads terraform.tfstate and writes resources.json (the direct-engine state file).
databricks bundle deployment migrate --target <env>

# 3. Verify with a read-only plan. It must show 0 add / 0 change / 0 delete.
databricks bundle plan --target <env>

# 4. Finalize. This step is what persists the switch to the workspace.
databricks bundle deploy --target <env>

Enable the engine with a top-level key in databricks.yml:

bundle:
  name: my_bundle
  engine: direct

You can also set it per target with targets.<env>.engine: direct, or via the DATABRICKS_BUNDLE_ENGINE=direct environment variable.

Points confirmed by the official documentation:

• bundle deployment migrate is local only. It writes resources.json (gitignored) and reads existing state IDs. Nothing changes server-side until step 4 (deploy).
• bundle plan is read-only: it "displays the actions … without making any changes." This is the correct dry run.
• The plan may show small metadata-only differences even when your configuration matches what is deployed. This is because the old Terraform state stored some fields that Databricks fills in automatically and that your bundle configuration does not include. These differences are harmless and are resolved on the next deploy.
• Requires CLI 1.3.0 or later (direct engine GA). Use the latest 1.3.x or newer where possible.
• Direct-engine-only resource types (not deployable under Terraform): genie_spaces, catalogs, external_locations, and Vector Search endpoints/indexes.
• Migration is permanent once completed. Do not re-run migrate on an already-migrated target; it will error.

Rolling back before step 4: delete resources.json and restore terraform.tfstate from the .backup file that migrate created (migrate prints the exact paths). The workspace is unaffected until you deploy.

When is bind required?

For a straightforward migration, bind is not required. migrate converts the state in place, and every resource already managed by the bundle is carried over automatically. If you are only switching an existing bundle from Terraform to direct and changing nothing else, you can skip bind and proceed to the cutover procedure.

bind is a separate, optional step that adopts a resource not yet present in the bundle's state, so that DABs manages it without recreating it. The common case during a migration is adopting a resource that was previously managed outside the bundle, for example, a Genie space maintained by a custom API script, or a dashboard or job created manually in the UI. You bind the resource defined in your bundle to the live resource's ID so it is adopted rather than recreated. This is the worked example below.

When you do this, run bind as a deliberate cutover step, never as a quick test (see Item 1).

Behaviors worth knowing before you migrate

Each item notes briefly how it was established during this migration.

1. bundle deployment bind persists to remote state before any deploy

Seen in a customer environment.

This item applies only when you are adopting a resource with bind (see "When is bind required?" above). A plain migration does not use bind.

bundle deployment bind KEY RESOURCE_ID links an existing workspace resource to one defined in your bundle, so that DABs manages it without recreating it. The command is documented on the Migrate existing resources to a bundle page. A detail worth calling out is its timing: bind writes to the workspace's deployment state immediately, before you run deploy. Because of this, running migrate and then bind against a shared target is not a safe way to "test" a migration; it changes real state.

What happened in practice: running migrate → bind → plan against production as a test left the state ahead of the configuration, and a later routine deployment was blocked by the safety gate (Item 2). The Case study below describes this incident in full.

Guidance:

• Treat this timing as observed behavior, verified during the migration.
• The only read-only command is bundle plan; it is the only safe way to preview a migration. When you genuinely need to adopt a resource, run bind as part of a real cutover (migrate → bind → plan → deploy), never against a shared or production target as a test.

2. The destructive-action gate aborts the entire deployment

Under the direct engine, if a bundle deploy would delete or recreate any resource, the CLI aborts the entire deployment (not only the affected resource) unless an explicit bypass flag is passed. In non-interactive CI, it fails with: "the deployment requires destructive actions, but the current console does not support prompting."

• This is a safety mechanism. If one resource is set to be deleted or recreated, it blocks all other resources (jobs, pipelines, and so on) from deploying too. The benefit is that nothing is deleted by surprise; the cost is that one unexpected deletion can block the whole deployment.
• The CLI prints a note advising automated agents not to retry with the bypass flag. Fix the underlying cause rather than bypassing the gate. During a migration, this gate is what protects you if a bind step (Item 1) left the state and configuration out of sync.

3. Run migrate before bind

When adopting a direct-engine-only resource (such as a Genie space) during migration, run migrate first. Running bundle deployment bind while the bundle is still on the Terraform engine fails with no converter for resource type genie_spaces. This is not specific to Genie spaces; it happens for any direct-engine-only resource type added to a Terraform-engine bundle, because these types have no converter under the Terraform provider. Run migrate (which converts the state to direct) and then bind.

4. A service principal cannot own a Genie space

There is no IS_OWNER permission level for genie_spaces and no ownership-transfer path for them (unlike dashboards, queries, and alerts). If your bundle deploys as a service principal, the Genie space must retain a human owner. Grant the deploy service principal CAN_MANAGE so that DABs can update the space's content. Note that run_as on a Genie space is an execution setting, not an ownership assignment.

5. Bundles that deploy as a service principal with run_as must be migrated in CI

A bundle with run_as: <service-principal> and any dashboards using embed_credentials: true will fail local validation for a human user (… do not support a run_as user different from the owner), because your local identity differs from the run_as service principal. CI succeeds because it runs as the service principal (the running identity matches run_as). So the only reliable way to preview the migration for such a bundle is a read-only bundle plan run in CI as the service principal, for example, a plan-only workflow on a temporary branch. Do not run migrate, bind, or deploy in that preview workflow.

Example: adopting a Genie space into the bundle

Native Genie support is one of the primary benefits of the direct engine, so it is worth a complete walkthrough. genie_spaces is a direct-engine-only resource type; it does not exist under the Terraform engine. Before the direct engine, a Genie space could only be managed outside the bundle.

Before: Genie space managed by an API script

Teams typically maintained the space with a custom deployment script that called the Genie REST API on each change, together with a JSON representation of the space definition checked into the repository:

scripts/deploy_genie_space.py      # custom code that PATCHes the space via the /genie/spaces API
src/genie_space_definition.json    # the space content, maintained by hand

This approach has several drawbacks: the space is managed outside the bundle (a separate CI step with its own authentication and its own way of failing), the JSON format is custom, there is no plan step to preview changes, and the space is not deployed together with the dashboards and jobs it relates to.

After: Genie space as a native genie_spaces resource

Once the bundle is on the direct engine, generate the resource from the existing live space so that nothing is recreated:

# Generates resources/<key>.genie_space.yml and a serialized <key>.geniespace.json content file.
databricks bundle generate genie-space \
  --existing-id <existing_genie_space_id> \
  --key my_genie_space

This produces a resource definition similar to the following:

# resources/my_genie_space.genie_space.yml
resources:
  genie_spaces:
    my_genie_space:
      title: "My Genie Space"
      warehouse_id: ${var.warehouse_id}
      parent_path: /Workspace/Shared/my_project
      # file_path points at the generated serialized content.
      file_path: ../src/my_genie_space.geniespace.json
      permissions:
        # A service principal cannot own a Genie space (Item 4). Keep a human owner
        # and grant the deploy service principal CAN_MANAGE so DABs can update content.
        - level: CAN_MANAGE
          service_principal_name: <deploy-service-principal>
        - level: CAN_MANAGE
          user_name: <human-owner>

Adopt the existing space rather than recreating it

The objective is to take over the live space, including its conversation history, rather than create a new one. Once the resource exists in configuration:

# 1. Migrate the bundle to direct first. bind fails under Terraform with
#    "no converter for resource type genie_spaces" (Item 3).
databricks bundle deployment migrate --target <env>

# 2. Adopt the live space into the bundle state by its ID.
databricks bundle deployment bind my_genie_space <existing_genie_space_id> --target <env>

# 3. Run a read-only plan. The space should appear as an in-place update, not a create + delete.
databricks bundle plan --target <env>

# 4. Deploy to finalize (in CI as the service principal for run_as bundles).
databricks bundle deploy --target <env>

A clean adoption reports the space as an in-place update with its ID preserved, for example, 1 to add (the permissions block), 2 to change, 0 to delete, with everything else unchanged. The space and its conversation history are retained.

Note: as in Item 1, this bind persists to remote state, so preview with a plan first and run migrate → bind → deploy only as a deliberate cutover, never as a test against production.

Retire the previous approach

Once the DAB-managed space deploys cleanly and you have verified the title, ID, and queries, remove the previous deploy_genie_space.py script and the legacy JSON file in a follow-up change. Retain them until the DAB-managed path is proven, then remove them so that there is a single source of truth.

Result: the Genie space now resides in the same bundle as the dashboards and jobs that feed it, receives the same plan, review, and CI treatment as every other resource, and no longer requires a separate API script.

Recommended production cutover procedure

The production cutover follows the standard procedure above, with two production-specific points. Deploy in CI as the service principal for run_as bundles, and run bind only for a resource you are genuinely adopting.

Gate before deploying: the plan must show 0 delete / 0 recreate. Metadata-only differences (see the standard procedure) are acceptable. Any create + delete or recreate means something is wrong; stop, do not deploy, and fix the bind step first.

After the deploy, verify the result: the Genie space ID is unchanged, the apps are still running, and no service principal was changed.

Do not run migrate, bind, or deploy against production as a test. Preview with bundle plan only.

Case study

I was migrating a single large bundle (jobs, dashboards, a Genie space, and two Apps, deploying as a service principal). To make sure the production cutover would be safe, I ran migrate → bind → plan against production first, as a test.

The plan looked clean. But the bind step had already changed production's state: it switched production to the direct engine and added the Genie space to the state, even though the configuration that defined the Genie space was still in an unmerged pull request. A few days later, my colleague's normal deployment ran. It compared the configuration (no Genie space) against the state (Genie space present), decided to delete it, hit the safety gate, and blocked all production deployments; not just Genie changes. Nothing was actually deleted (the gate stopped it, and the space and its history were safe), but production stayed blocked until the pull request was merged.

Root cause: treating migrate and bind as a test. Lesson: the only safe way to preview a migration is bundle plan. bind changes state on its own, without a deploy. A clean plan only tells you the plan step succeeded; the bind before it had already changed the state.

Summary checklist

• [ ] CLI 1.3.0 or later (ideally the latest 1.3.x or newer).
• [ ] Add engine: direct; migrate one target at a time, lowest-risk environment first.
• [ ] Use bundle plan as the only way to preview. Do not run migrate or bind against a shared or production target to test.
• [ ] For service-principal bundles, run migrate, plan, and deploy in CI as the service principal, not locally.
• [ ] Gate on 0 delete / 0 recreate before deploying (metadata-only differences are acceptable).
• [ ] For a Genie space, generate the resource from the existing ID, then migrate → bind → plan → deploy to adopt it in place; keep a human owner and grant the deploy service principal CAN_MANAGE; retire the previous API script afterward.
• [ ] After cutover, verify the Genie space ID and apps are unchanged.

Sources and further reading

• Official documentation: Migrate to the direct deployment engine · bundle command group · Migrate existing resources to a bundle · What's coming · Bundles release notes
• Internal engineering design documents exist for the engine internals. At the time of writing, there is no field-authored generic migration runbook, which is the motivation for this guide. Several of the behaviors above come from hands-on migration experience.

[Armanillo_RoK]

Hi @himanshu_ranjan,
 
thanks for the heads-up!

Genie Agents (still genie_spaces as a bundle resource type) landed in bundles because the Terraform provider has no equivalent resource, and the direct engine removed the requirement to wait for a provider release. Fair enough.

But that decoupling means the set of bundle-deployable resources will now grow faster than the provider's, and it has already reached infrastructure-shaped objects, namely catalogs and external locations are direct-engine bundle resources today, and those sit squarely in most platform teams' Terraform.

So is there a stated principle for which layer owns what going forward, or does it land wherever ships first?

Specifically, if the provider eventually adds a Genie resource, is the guidance to move or to stay in bundles? My own view is stay in DABs since a Genie Agent isn't really infrastructure, it's instructions, example queries and trusted assets that change with the data model, so it belongs next to the pipelines it describes. But I'd rather know if that's the intended direction.

And for a resource both layers can manage, is there anything preventing two tools from claiming the same object, or is that purely down to team discipline?

[himanshu_ranjan]

Thanks @Armanillo_RoK for reading it closely, and glad it was useful. You've read the decoupling correctly: because the direct engine no longer waits on a Terraform provider release, the set of bundle-supported resources can grow on its own, and it now reaches infrastructure-shaped objects like catalogs and external locations.

On which layer owns what, there's actually a stated recommendation, not just "wherever ships first." The Databricks deployment guide (the IaC section) suggests Terraform for infrastructure like workspaces, networking, Unity Catalog and storage, and Bundles for data and AI workloads, where it lists jobs, pipelines, notebooks and models:
https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/iac

Being able to deploy catalogs and external locations from a bundle is a new capability, but it doesn't move that recommendation. By default those still sit with the platform team's Terraform. I'd only manage one from a bundle as a deliberate, agreed exception. For example, a self-contained catalog that belongs to a single workload, where the platform team has decided to leave it to that team instead of managing it centrally.

On Genie, I'd keep it in the bundle, for the reason you gave. A Genie space isn't really infrastructure, it's instructions, sample questions and trusted assets that move with the data model, so it belongs next to the pipelines it describes. Terraform provider support for Genie is an open feature request, not something available today, and even if it lands I'd let ownership decide rather than the fact that a new resource exists. A Genie space tied to a data product stays in the bundle.

On two tools managing the same object, there isn't a cross-tool safeguard. Bundles on the direct engine keep their own deployment state and Terraform keeps its own state file, and neither one knows about the other, so each can think it's authoritative. bundle deployment bind and Terraform's import are explicit ways to adopt an existing resource, not conflict detectors. So the real guardrail is ownership: one authoritative tool per object, usually split by team, with the platform team owning the Terraform repo and the data engineers owning the bundle, and the bundle referencing infra objects by name instead of redeclaring them. When you do move ownership between the two, do it deliberately and check with a plan that shows no unintended create, replace or delete. That's why I lean on bundle plan as the safe preview in the post.

[EduardoLomonaco]

Great post, I wasn't aware of the change and helps me learn about it. Will greatly benefit one of my customers!

[-werners-]

Tnx, helpful guide!

However I do notice that the direct engine still lacks some functionality.
fetching values from cluster policies apparently is only partially implemented.
I had Genie make me a nice summary :-):

Summary: After migrating from the Terraform engine to the direct engine, jobs with apply_policy_default_values: true are flagged as "changed" on every bundle plan / bundle deploy, even when no YAML changes were made. This creates an infinite update loop because the cluster policy re-injects the same values after each deploy.

Current behavior: The direct engine correctly skips some policy/platform-populated fields:

• azure_attributes → "action": "skip", "reason": "managed"
• data_security_mode → "action": "skip", "reason": "backend_default"
• enable_elastic_disk → "action": "skip", "reason": "backend_default"
• node_type_id → "action": "skip", "reason": "backend_default"

But it flags these policy-injected fields as "action": "update":

• spark_env_vars
• spark_version
• spark_conf
• custom_tags (e.g. ResourceClass: SingleNode)
• runtime_engine

Expected behavior: When apply_policy_default_values: true is set on a job cluster, the engine should recognize that the policy is the source of truth for fields not declared in the bundle YAML. Fields present in remote state but absent from local config — that are injected by the referenced policy_id — should be treated as "action": "skip", "reason": "managed" (or a new reason like "policy_default"), not as removals.

[himanshu_ranjan]

Thanks @-werners-, glad the guide helped.

Your question is a bit outside what the blog covers. What you are seeing is a different thing: how the direct engine compares the values your cluster policy fills in each time you plan or deploy.

It's a known issue and it is being worked on. A recent CLI fix made the engine leave apply_policy_default_values, data_security_mode, enable_elastic_disk, node_type_id and a few others alone, which is why you see those skipped. The ones you listed, spark_conf, spark_version, spark_env_vars, runtime_engine and custom_tags, are not handled yet, so they still show up as an update every time.

A quick workaround is to set those fields yourself in the bundle YAML. Then your config matches what the policy adds and the diff goes away. You lose a little of the benefit of the policy, but it stops the constant changes until the engine catches up.