article DABs Migration Guide: Terraform to the Direct Deployment Engine in Technical Blog

DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Tue, 18 Aug 2026 09:25:25 GMT

A guide to migrate an existing Declarative Automation Bundle from the Terraform deployment engine to the direct deployment engine. It covers the standard procedure, when bundle deployment bind is required, and five behaviours that surfaced during a customer migration, and a worked example of adopting a Genie space into the bundle.

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

Armanillo_RoK — Thu, 20 Aug 2026 13:22:11 GMT

Hi @himanshu_ranjan,

thanks for the heads-up!

Genie Agents (still genie_spaces as a bundle resource type) landed in bundles because the Terraform provider has no equivalent resource, and the direct engine removed the requirement to wait for a provider release. Fair enough.

But that decoupling means the set of bundle-deployable resources will now grow faster than the provider's, and it has already reached infrastructure-shaped objects, namely catalogs and external locations are direct-engine bundle resources today, and those sit squarely in most platform teams' Terraform.

So is there a stated principle for which layer owns what going forward, or does it land wherever ships first?

Specifically, if the provider eventually adds a Genie resource, is the guidance to move or to stay in bundles? My own view is stay in DABs since a Genie Agent isn't really infrastructure, it's instructions, example queries and trusted assets that change with the data model, so it belongs next to the pipelines it describes. But I'd rather know if that's the intended direction.

And for a resource both layers can manage, is there anything preventing two tools from claiming the same object, or is that purely down to team discipline?

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Thu, 20 Aug 2026 17:40:22 GMT

Thanks @Armanillo_RoK for reading it closely, and glad it was useful. You've read the decoupling correctly: because the direct engine no longer waits on a Terraform provider release, the set of bundle-supported resources can grow on its own, and it now reaches infrastructure-shaped objects like catalogs and external locations.

On which layer owns what, there's actually a stated recommendation, not just "wherever ships first." The Databricks deployment guide (the IaC section) suggests Terraform for infrastructure like workspaces, networking, Unity Catalog and storage, and Bundles for data and AI workloads, where it lists jobs, pipelines, notebooks and models:
https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/iac

Being able to deploy catalogs and external locations from a bundle is a new capability, but it doesn't move that recommendation. By default those still sit with the platform team's Terraform. I'd only manage one from a bundle as a deliberate, agreed exception. For example, a self-contained catalog that belongs to a single workload, where the platform team has decided to leave it to that team instead of managing it centrally.

On Genie, I'd keep it in the bundle, for the reason you gave. A Genie space isn't really infrastructure, it's instructions, sample questions and trusted assets that move with the data model, so it belongs next to the pipelines it describes. Terraform provider support for Genie is an open feature request, not something available today, and even if it lands I'd let ownership decide rather than the fact that a new resource exists. A Genie space tied to a data product stays in the bundle.

On two tools managing the same object, there isn't a cross-tool safeguard. Bundles on the direct engine keep their own deployment state and Terraform keeps its own state file, and neither one knows about the other, so each can think it's authoritative. bundle deployment bind and Terraform's import are explicit ways to adopt an existing resource, not conflict detectors. So the real guardrail is ownership: one authoritative tool per object, usually split by team, with the platform team owning the Terraform repo and the data engineers owning the bundle, and the bundle referencing infra objects by name instead of redeclaring them. When you do move ownership between the two, do it deliberately and check with a plan that shows no unintended create, replace or delete. That's why I lean on bundle plan as the safe preview in the post.

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

EduardoLomonaco — Fri, 21 Aug 2026 02:20:21 GMT

Great post, I wasn't aware of the change and helps me learn about it. Will greatly benefit one of my customers!

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

-werners- — Fri, 21 Aug 2026 08:47:24 GMT

Tnx, helpful guide!

However I do notice that the direct engine still lacks some functionality.
fetching values from cluster policies apparently is only partially implemented.
I had Genie make me a nice summary :-):

Summary: After migrating from the Terraform engine to the direct engine, jobs with apply_policy_default_values: true are flagged as "changed" on every bundle plan / bundle deploy, even when no YAML changes were made. This creates an infinite update loop because the cluster policy re-injects the same values after each deploy.

Current behavior: The direct engine correctly skips some policy/platform-populated fields:

azure_attributes → "action": "skip", "reason": "managed"
data_security_mode → "action": "skip", "reason": "backend_default"
enable_elastic_disk → "action": "skip", "reason": "backend_default"
node_type_id → "action": "skip", "reason": "backend_default"

But it flags these policy-injected fields as "action": "update":

spark_env_vars
spark_version
spark_conf
custom_tags (e.g. ResourceClass: SingleNode)
runtime_engine

Expected behavior: When apply_policy_default_values: true is set on a job cluster, the engine should recognize that the policy is the source of truth for fields not declared in the bundle YAML. Fields present in remote state but absent from local config — that are injected by the referenced policy_id — should be treated as "action": "skip", "reason": "managed" (or a new reason like "policy_default"), not as removals.

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Fri, 21 Aug 2026 21:01:06 GMT

Thanks @-werners-, glad the guide helped.

Your question is a bit outside what the blog covers. What you are seeing is a different thing: how the direct engine compares the values your cluster policy fills in each time you plan or deploy.

It's a known issue and it is being worked on. A recent CLI fix made the engine leave apply_policy_default_values, data_security_mode, enable_elastic_disk, node_type_id and a few others alone, which is why you see those skipped. The ones you listed, spark_conf, spark_version, spark_env_vars, runtime_engine and custom_tags, are not handled yet, so they still show up as an update every time.

A quick workaround is to set those fields yourself in the bundle YAML. Then your config matches what the policy adds and the diff goes away. You lose a little of the benefit of the policy, but it stops the constant changes until the engine catches up.