How to change a field when instancing cluster defined as variable?

yit337 — Tue, 09 Jun 2026 14:39:00 GMT

I define all clusters as variable in separate files, so I can re-use them.

Then I am accessing them in jobs as:

The issue is that I want to change just the custom_tags in the cluster when instancing it for a job, cause my tags are different for each job. Could this be achieved somehow ?

Re: How to change a field when instancing cluster defined as variable?

ShamenParis — Tue, 09 Jun 2026 16:47:14 GMT

Yes, you can achieve this seamlessly, but not by overriding the custom_tags inside the cluster variable. Instead, you define your specific tags at the Job level, and Databricks automatically merges them with your cluster variable's tags.

Because complex variables in Databricks Asset Bundles (DABs) are substituted entirely as a block, DABs do not support deep-merging or partial overrides (like just changing custom_tags) at the interpolation site (DABs Variables Documentation).

When you define tags on the job itself, Databricks' built-in tag inheritance takes over:

Tag Propagation: Any tags you define at the Job level automatically propagate down to all Job Clusters created by that job.

Merging: Databricks safely combines the Job-level tags with any Cluster-level tags defined inside your reusable cluster variable.

Overwrite Precedence: If both the job and the cluster variable contain a tag with the exact same key, the Cluster-level tag takes precedence (it overwrites the job tag).

Best Practice: Put static, company-wide tags (e.g., environment, department) inside your reusable cluster variable. Put dynamic, job-specific tags (e.g., pipeline_name, job_owner) directly in the job YAML.

Example YAML:

jobs:
  my_specific_job:
    name: "My Job Name"
    # 1. Job-level tags (these propagate to the cluster automatically)
    tags:
      custom_job_tag: "unique_value_for_this_job" 
    
    job_clusters:
      - job_cluster_key: "default_cluster"
        # 2. Reusable cluster variable (with base tags)
        new_cluster: ${var.my_base_cluster}

Databricks Tag Propagation & Precedence: Read more in the Databricks Usage Details Tags Documentation (Under "How tags propagate" and "Custom tags").

topic How to change a field when instancing cluster defined as variable? in Data Engineering

How to change a field when instancing cluster defined as variable?

Re: How to change a field when instancing cluster defined as variable?