Yes, you can achieve this seamlessly, but not by overriding the custom_tags inside the cluster variable. Instead, you define your specific tags at the Job level, and Databricks automatically merges them with your cluster variable's tags.
Because complex variables in Databricks Asset Bundles (DABs) are substituted entirely as a block, DABs do not support deep-merging or partial overrides (like just changing custom_tags) at the interpolation site (DABs Variables Documentation).
When you define tags on the job itself, Databricks' built-in tag inheritance takes over:
Tag Propagation: Any tags you define at the Job level automatically propagate down to all Job Clusters created by that job.
Merging: Databricks safely combines the Job-level tags with any Cluster-level tags defined inside your reusable cluster variable.
Overwrite Precedence: If both the job and the cluster variable contain a tag with the exact same key, the Cluster-level tag takes precedence (it overwrites the job tag).
Best Practice: Put static, company-wide tags (e.g., environment, department) inside your reusable cluster variable. Put dynamic, job-specific tags (e.g., pipeline_name, job_owner) directly in the job YAML.
Example YAML:
jobs:
my_specific_job:
name: "My Job Name"
# 1. Job-level tags (these propagate to the cluster automatically)
tags:
custom_job_tag: "unique_value_for_this_job"
job_clusters:
- job_cluster_key: "default_cluster"
# 2. Reusable cluster variable (with base tags)
new_cluster: ${var.my_base_cluster} Databricks Tag Propagation & Precedence: Read more in the Databricks Usage Details Tags Documentation (Under "How tags propagate" and "Custom tags").
