https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/ba-p/124772 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <H1>&nbsp;</H1> <H1><SPAN>Introduction</SPAN></H1> <P><SPAN>Databricks Asset Bundles (DAB) is a structured way to define, deploy, and manage Databricks workflows, including jobs, clusters, dashboards, model serving endpoints, and other resources, using declarative YAML configurations. It allows us to implement software engineering best practices by enabling version control, CI/CD integration, and automation by treating infrastructure and data assets as code. DAB simplifies collaboration and deployment across environments, ensuring easier management and scalability of Databricks projects.</SPAN></P> <P class="lia-indent-padding-left-30px"><EM>If you are completely new to DAB, please refer to <A href="https://docs.databricks.com/aws/en/dev-tools/bundles/" target="_blank" rel="noopener">What are Databricks Asset Bundles</A> before continuing with this blog post.</EM></P> <P><SPAN>By using DAB, you can define Databricks resources like jobs, pipelines, and notebooks as source files. These files fully describe a project and the code that governs it, providing project structure and automation for testing and deployment. Additionally, DAB allows us to define deployment targets that can be fully customized depending on the use case and the project’s needs, such as development, staging, and production.</SPAN></P> <P><SPAN>In some cases, specific environments may require additional resources. For example, staging might include extra testing pipelines that are not needed in production once validation is complete.</SPAN></P> <P><SPAN>In this blog post, we will explore different ways of customizing resource deployments per target.</SPAN></P> <P>&nbsp;</P> <H1><SPAN>Using <FONT face="andale mono,times">include</FONT> and <FONT face="andale mono,times">resources</FONT> mapping</SPAN></H1> <P><SPAN>As we know, Databricks resources are defined by specifying the type of resource and its configuration under the </SPAN><FONT face="andale mono,times"><SPAN>resources </SPAN></FONT><SPAN>mapping in </SPAN><FONT face="andale mono,times"><SPAN>databricks.yml</SPAN></FONT><SPAN>. Resources can include Databricks apps, clusters, dashboards, jobs, pipelines, model serving endpoints, and more. The following example illustrates how to define a simple job and a job cluster:</SPAN></P> <LI-CODE lang="markup">resources: jobs: hello-job: name: hello-job max_concurrent_runs: 1 job_clusters: - job_cluster_key: small_cluster new_cluster: spark_version: 16.4.x-scala2.13 node_type_id: m5d.large num_workers: 1 tasks: - task_key: hello-task notebook_task: notebook_path: ./hello.py </LI-CODE> <P><SPAN>By default, resources can be declared at the top level, making them available across all deployment targets:</SPAN></P> <LI-CODE lang="markup"># databricks.yml bundle: name: test-bundle resources: ... targets: dev: default: true ... stg: ... prd: ...</LI-CODE> <P><SPAN>With this setup, the same resources are deployed to dev, stg, and prd, ensuring consistency across all environments.</SPAN></P> <P><SPAN>In many cases, different environments require unique configurations. For example, you might need additional testing pipelines in staging but not in production. To achieve this, you can define resources specific to each deployment target:</SPAN></P> <LI-CODE lang="markup"># databricks.yml bundle: name: test-bundle resources: ... targets: dev: default: true resources: ... ... stg: resources: ... ... prd: resources: ... ...</LI-CODE> <P><SPAN>By defining resources at the target level, additional resources are deployed only where needed, while still inheriting the global resources from the top level. This flexible approach ensures that each environment is optimized for its purpose without unnecessary configurations.</SPAN></P> <P><SPAN>Be aware that each resource has an </SPAN><STRONG>identifier</STRONG><SPAN>. If you use the same identifier at both the top level and the target level, the target-level definition will take precedence and override the top-level definition.&nbsp;</SPAN></P> <P class="lia-indent-padding-left-30px"><EM>For detailed guidance on customizing configurations for specific targets, refer to the <A href="https://docs.databricks.com/aws/en/dev-tools/bundles/cluster-override" target="_blank" rel="noopener">Override cluster settings</A> section in the Databricks Asset Bundles documentation. Additionally, explore a <A href="https://github.com/alexott/dabs-playground/tree/main/integration-tests" target="_blank" rel="noopener">sample code</A> that illustrates how to configure integration tests by redefining resources for each target environment.</EM></P> <P><SPAN>So far, everything has been defined in a single YAML file, which can make readability and management challenging as the project grows. Let’s explore some strategies to enhance flexibility, support customizations, and ensure seamless scalability as the project expands.</SPAN></P> <P><SPAN>The </SPAN><FONT face="andale mono,times"><SPAN>include</SPAN></FONT><SPAN> mapping allows us to add a list of path globs containing configuration files to include within the bundle. These path globs are relative to the location of the bundle configuration file in which the path globs are specified.</SPAN></P> <P><SPAN>Therefore, we can structure the project in the following form:</SPAN></P> <LI-CODE lang="markup">project/ ├── tests/ │ └── ... ├── resources/ │ ├── pipelines.yml │ ├── jobs.yml │ └── dashboards.yml ├── src/ │ ├── notebook_a.ipynb │ ├── notebook_b.ipynb │ └── ... ├── databricks.yml └── ...</LI-CODE> <P><SPAN>And instead of using resources in a single YAML, we just specify what we want to include:</SPAN></P> <LI-CODE lang="markup"># databricks.yml bundle: name: test-bundle include: - resources/*.yml targets: dev: default: true ... stg: ... prd: ...</LI-CODE> <P><SPAN>Each file in </SPAN><FONT face="andale mono,times"><SPAN>resources/*.yml</SPAN></FONT><SPAN> contains its own resource definition, allowing for a structured and modular approach to managing YAML files. This separation keeps resource definitions organized and easy to manage as the project scales. However, the </SPAN><FONT face="andale mono,times"><SPAN>include</SPAN></FONT><SPAN> mapping can only be used at the top level, meaning all included resources will be deployed to every target.</SPAN></P> <P class="lia-indent-padding-left-30px"><EM>The important fact to note here is that the include mapping reads all the files, combining the resource definitions and merging them per target, allowing overrides as mentioned above.</EM></P> <P><SPAN>If we need to add resources for a specific target, we must define resources at the target level, as shown in the previous example. While this approach works, it can become cumbersome when managing numerous customizations across multiple targets. To address this challenge, let’s explore a better way of structuring target-specific YAML files in the next section.</SPAN></P> <P>&nbsp;</P> <H1><SPAN>Separating Target YAMLs</SPAN></H1> <P><SPAN>A key advantage of DAB is that the </SPAN><FONT face="andale mono,times"><SPAN>include</SPAN></FONT><SPAN> directive is not limited to resources - it can also be used for other top-level keys like </SPAN><FONT face="andale mono,times"><SPAN>targets </SPAN></FONT><SPAN>mapping. This allows for a more modular and scalable project structure.</SPAN></P> <P><SPAN>To improve organization and maintainability, we can structure our project as follows:</SPAN></P> <LI-CODE lang="markup">project/ ├── tests/ │ └── ... ├── resources/ │ ├── pipelines.yml │ ├── jobs.yml │ └── dashboards.yml ├── src/ │ ├── notebook_a.ipynb │ ├── notebook_b.ipynb │ └── ... ├── targets/ │ ├── dev.yml │ ├── stg.yml │ └── prd.yml ├── databricks.yml └── ... </LI-CODE> <P><SPAN>In this setup:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><FONT face="andale mono,times"><SPAN>resources/</SPAN></FONT><SPAN> directory contains shared resources used by all environments.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><FONT face="andale mono,times"><SPAN>targets/</SPAN></FONT><SPAN> directory holds YAML files that define the specific resources for each deployment target.</SPAN></LI> </UL> <P><SPAN>Each target file, such as </SPAN><FONT face="andale mono,times"><SPAN>targets/dev.yml</SPAN></FONT><SPAN>, includes only the resources specific to that environment.</SPAN></P> <LI-CODE lang="markup"># targets/dev.yml targets: dev: default: true resources: ... ...</LI-CODE> <P><SPAN>Similarly, </SPAN><FONT face="andale mono,times"><SPAN>targets/stg.yml</SPAN></FONT><SPAN> and </SPAN><FONT face="andale mono,times"><SPAN>targets/prd.yml</SPAN></FONT><SPAN> will include the appropriate resources for their respective environments.</SPAN></P> <P><SPAN>In </SPAN><FONT face="andale mono,times"><SPAN>databricks.yml</SPAN></FONT><SPAN>, we can now include the common resources while allowing each target to bring its own specific configurations:</SPAN></P> <LI-CODE lang="markup"># databricks.yml bundle: name: test-bundle include: - resources/*.yml - targets/*.yml</LI-CODE> <P><SPAN>This setup provides a structured and scalable way to manage resources, ensuring that each environment gets precisely the resources it needs without unnecessary duplication. By splitting resource definitions across separate YAML files for each target, project teams gain better organization, control, and flexibility.</SPAN></P> <P><SPAN>This modular approach simplifies configuration management, making it easier to track changes, customize deployments, and avoid bloated YAML files.</SPAN></P> <P>&nbsp;</P> <H1><SPAN>Splitting Resources</SPAN></H1> <P><SPAN>The previous examples grouped multiple resources into shared YAML files. Another approach is to define resources individually and associate them directly with their target environments. This creates a clean, intuitive structure where each resource lives next to the target it’s meant for.</SPAN></P> <P><SPAN>Example project layout:</SPAN></P> <LI-CODE lang="markup">project/ ├── tests/ │ └── ... ├── resources/ │ ├── pipeline_a.yml │ ├── pipeline_b.yml │ └── ... ├── src/ │ ├── notebook_a.ipynb │ ├── notebook_b.ipynb │ └── ... ├── databricks.yml └── ...</LI-CODE> <P><SPAN>In this setup, </SPAN><FONT face="andale mono,times"><SPAN>databricks.yml</SPAN></FONT><SPAN> includes all resource files:</SPAN></P> <LI-CODE lang="markup"># databricks.yml bundle: name: test-bundle include: - resources/*.yml targets: dev: ... stg: ... prd: ...</LI-CODE> <P><SPAN>Each resource file defines a </SPAN><FONT face="andale mono,times"><SPAN>resources</SPAN></FONT><SPAN> mapping alongside the target(s) it’s intended for. For example:</SPAN></P> <LI-CODE lang="markup"># pipeline_a.yml anchor_name: &amp;anchor_name resources: ... targets: dev: &lt;&lt;: *anchor_name </LI-CODE><LI-CODE lang="markup"># pipeline_b.yml anchor_name: &amp;anchor_name resources: ... targets: dev: &lt;&lt;: *anchor_name stg: &lt;&lt;: *anchor_name</LI-CODE> <P><SPAN>Here we are leveraging </SPAN><A href="https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/" target="_blank" rel="noopener"><SPAN>YAML anchors</SPAN></A><SPAN> to keep the resource definition DRY and easy to read. YAML anchors are a feature that allows you to define reusable blocks of configuration, which can then be referenced elsewhere in your YAML file to avoid repetition.</SPAN></P> <P><SPAN>The </SPAN><FONT face="andale mono,times"><SPAN>pipeline_a.yml</SPAN></FONT><SPAN> will only deploy its defined resources to the </SPAN><SPAN>dev</SPAN><SPAN> environment, while the </SPAN><SPAN>pipeline_b.yml</SPAN><SPAN> will deploy to both the </SPAN><SPAN>dev</SPAN><SPAN> and </SPAN><SPAN>stg</SPAN><SPAN> environments. The main </SPAN><FONT face="andale mono,times">databricks.yml</FONT><SPAN> defines the targets, and the included YAML file extends the corresponding section.</SPAN></P> <P><SPAN>You can freely mix this approach with the previously described structures. Functionally, they achieve the same result. The choice depends on your preferences and the level of modularity you need. This method is especially powerful when you have many environment-specific resources and want to avoid cluttering large files with multiple unrelated configurations.</SPAN></P> <P class="lia-indent-padding-left-30px"><EM>Explore the following <A href="https://github.com/alexott/dabs-playground/tree/main/mlops-statcks-multiphase" target="_blank" rel="noopener">code example</A> to see how this approach enables the implementation of a comprehensive MLOps project, from training and validation to deployment, using a multiphase workflow with customized resources tailored for each stage.</EM></P> <P class="lia-indent-padding-left-30px">&nbsp;</P> <H1>Runtime Editing</H1> <P><SPAN>DAB supports </SPAN><A href="https://docs.databricks.com/aws/en/dev-tools/bundles/variables" target="_blank" rel="noopener"><SPAN>substitutions and custom variables</SPAN></A><SPAN>, enabling modular, reusable, and dynamic configuration files. These features allow values to be retrieved at runtime, ensuring that resource configurations can be adjusted dynamically when deploying and running a bundle.</SPAN></P> <P><SPAN>Variables can be assigned different values for each target, and by leveraging default values, you can implement conditional overrides for specific resource settings as needed. Please refer to the following </SPAN><A href="https://github.com/alexott/dabs-playground/tree/main/vars_demo" target="_blank" rel="noopener"><SPAN>example code</SPAN></A><SPAN> to see how this logic can be implemented.</SPAN></P> <P><SPAN>Unfortunately, DAB does not currently support using variables to dynamically set </SPAN><FONT face="andale mono,times"><SPAN>include</SPAN></FONT><SPAN> directives. As a workaround, we can modify the </SPAN><FONT face="andale mono,times">databricks.yml</FONT><SPAN> file within a CI/CD pipeline by substituting variables before executing the </SPAN><FONT face="andale mono,times"><SPAN>databricks bundle deploy </SPAN></FONT><SPAN>command. This approach allows for greater flexibility in managing environment-specific configurations while maintaining automation in the deployment process.</SPAN></P> <P><SPAN>We can enhance our project structure by leveraging environment variables to dynamically set configurations for each target, making deployments even more flexible:</SPAN></P> <LI-CODE lang="markup">project/ ├── tests/ │ └── ... ├── resources/ | ├── common/ │ | ├── pipelines.yml │ | ├── jobs.yml │ | └── dashboards.yml | ├── dev/ │ | └── ... | ├── stg/ │ | └── ... | ├── prd/ │ | └── ... ├── src/ │ ├── notebook_a.ipynb │ ├── notebook_b.ipynb │ └── ... ├── targets/ │ ├── dev.yml │ ├── stg.yml │ └── prd.yml ├── databricks.yml └── ...</LI-CODE> <P><SPAN>Where <FONT face="andale mono,times">databricks.yml</FONT> looks like:</SPAN></P> <LI-CODE lang="markup"># databricks.yml bundle: name: test-bundle include: - resources/common/*.yml - resources/${target}/*/yml - targets/*.yml</LI-CODE> <P><SPAN>In a CI/CD pipeline, we typically pull the DAB code from a Git repository and set an environment variable that represents the target deployment environment. Before running the </SPAN><FONT face="andale mono,times"><SPAN>databricks bundle deploy </SPAN></FONT><SPAN>command, we can dynamically replace </SPAN><FONT face="andale mono,times"><SPAN>${target}</SPAN></FONT><SPAN> with the appropriate environment variable using a simple command like </SPAN><FONT face="andale mono,times"><SPAN>sed</SPAN></FONT><SPAN>:</SPAN></P> <LI-CODE lang="markup">sed -i -e 's/${target}/'"$TARGET"'/g' databricks.yml</LI-CODE> <P><SPAN>When running this command, every instance of </SPAN><FONT face="andale mono,times"><SPAN>${target}</SPAN></FONT><SPAN> in the </SPAN><FONT face="andale mono,times"><SPAN>databricks.yml</SPAN></FONT><SPAN> file is replaced with the value of the </SPAN><FONT face="andale mono,times"><SPAN>$TARGET</SPAN></FONT><SPAN> environment variable. This ensures that the Databricks bundle only loads the configuration and resources specific to the intended deployment environment.</SPAN></P> <P><SPAN>By leveraging this approach, we achieve greater flexibility, allowing each deployment to dynamically include the correct resources based on the target environment.</SPAN></P> <P>&nbsp;</P> <H1><SPAN>Conclusion</SPAN></H1> <P><SPAN>Scaling DAB projects and customizing deployments for multiple targets while maintaining a modular, manageable structure can be a challenge. However, with the right approach, you can streamline your workflow and reduce overhead as your projects grow.</SPAN></P> <P><SPAN>In this article, we explored several practical techniques to help you manage increasing complexity and customization needs.</SPAN></P> <P><SPAN>To recap:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN><STRONG>Use include and resource mapping</STRONG> - Instead of putting everything into a single </SPAN><SPAN>databricks.yml</SPAN><SPAN>, spread out resources into separate YAML files and use </SPAN><SPAN>include</SPAN><SPAN> mapping to assemble them. This improves clarity and scalability.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN><STRONG>Separating Target YAMLs</STRONG> - For environment-specific customization, maintain distinct YAML files for each target. This keeps configurations clean and easy to manage.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Splitting Resources</STRONG><SPAN>&nbsp;- Define each resource individually and associate them directly to their target environments in the same YAML. This modular approach makes updates and troubleshooting much simpler.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Runtime Editing</STRONG>&nbsp;<SPAN>- Utilize CI/CD pipelines and environment variables to dynamically adjust resource targets at deployment time, enabling flexible and automated workflows.</SPAN></LI> </UL> <P><SPAN>I hope these approaches and examples have provided you with a clearer understanding of how to structure your Databricks project using DAB. By implementing these strategies, you can achieve greater target customization while ensuring your deployment remains scalable, modular, and easy to manage.</SPAN></P> <P><SPAN>Happy coding!</SPAN></P> Mon, 21 Jul 2025 08:51:01 GMT Vladsiv 2025-07-21T08:51:01Z https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/ba-p/124772 <DIV class="border-borderMain/50 ring-borderMain/50 divide-borderMain/50 dark:divide-borderMainDark/50 dark:ring-borderMainDark/50 dark:border-borderMainDark/50 bg-transparent"> <DIV class="gap-y-sm md:gap-y-md flex flex-col"> <DIV class="relative font-sans text-base text-textMain dark:text-textMainDark selection:bg-super/50 selection:text-textMain dark:selection:bg-super/10 dark:selection:text-super"> <DIV class="min-w-0 break-words [word-break:break-word]"> <DIV id="markdown-content-0" class="gap-y-md after:clear-both after:block after:content-['']" dir="auto"> <DIV class="relative"> <DIV class="prose text-pretty dark:prose-invert inline leading-normal break-words min-w-0 [word-break:break-word]"> <P class="my-0">Discover how Databricks Asset Bundles (DAB) empowers teams to define, automate, and manage Databricks resources with ease using YAML-based configurations while at the same time enabling environment-specific customization, streamlining deployment and collaboration across multiple environments, from development to production.</P> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> Mon, 21 Jul 2025 08:51:01 GMT https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/ba-p/124772 Vladsiv 2025-07-21T08:51:01Z https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/bc-p/125298#M693 <P>Thanks for the blog! DABs are really a great tool to enhance the developer experience with Databricks.</P><P>I personally create a bunch of custom variables in my bundle and usually there are some that I won't change at run time, for example catalog, schema and others. These I'll keep in each target definition.</P><P>Then using Azure DevOps libraries (one per target environment) in which I can save safely authentication stuff (host, client_id, client_secret). These then I manage with Azure DevOps stages.</P><P>Finally, I have very few variables I want to modify when I use "databricks deploy" so I use variable override such:</P><PRE>databricks bundle validate --var="your_variable=new-value"</PRE><P>For example, I pass <STRONG>git_sha</STRONG> and <STRONG>git_branch</STRONG> for traceability which could be used as a tag for a job or a parameter in your package entrypoint.&nbsp;</P> Tue, 15 Jul 2025 12:36:37 GMT https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/bc-p/125298#M693 FedeRaimondi 2025-07-15T12:36:37Z https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/bc-p/131428#M757 <P>Hi, great post!</P><P>I am trying to dynamically change in the targets section the variables defined in the databricks.yml file. Eg:</P><PRE><BR /># databricks.yml<BR />variables:<BR /> catalog: <BR /> description: The Unity Catalog catalog to use.<BR /> default: my_catalog<BR /> schema:<BR /> description: The Unity Catalog schema basename to use as a prefix for all created schemas. bundle.target will be appended to this value.<BR /> default: my_schema<BR /> landing_data_volume:<BR /> description: The volume to use for landing data.<BR /> default: /Volumes/my_catalog/my_schema/landing_data</PRE><P>my ultimate goal is to use these variables in the targets section, in order to append a suffix such as "dev" to the schema to be deployed in the dev environment. However, a syntax like this:</P><PRE># databricks.yml</PRE><DIV><PRE><SPAN>targets</SPAN><SPAN>:</SPAN><BR /><SPAN> dev</SPAN><SPAN>:</SPAN><BR /><SPAN> mode</SPAN><SPAN>: </SPAN><SPAN>development</SPAN><BR /><SPAN> variables</SPAN><SPAN>:</SPAN><BR /><SPAN> catalog</SPAN><SPAN>: </SPAN><SPAN>${workspace.current_user.short_name}</SPAN><BR /><SPAN> schema</SPAN><SPAN>: </SPAN><SPAN>${var.schema}</SPAN><BR /><SPAN> default</SPAN><SPAN>: </SPAN><SPAN>true</SPAN><BR /><SPAN> workspace</SPAN><SPAN>:</SPAN><BR /><SPAN> host</SPAN><SPAN>: </SPAN><SPAN><A href="https://my_host.cloud.databricks.com" target="_blank">https://my_host.cloud.databricks.com</A></SPAN></PRE><DIV>&nbsp;</DIV><DIV><SPAN>results in the following error:</SPAN></DIV></DIV><PRE>Error: cycle detected in field resolution: variables.schema.default -&gt; var.schema -&gt; var.schema</PRE><P>which way would you recommend to sort this out?</P><P>Many thanks,</P><P>Domenico</P> Tue, 09 Sep 2025 16:04:57 GMT https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/bc-p/131428#M757 dsimone 2025-09-09T16:04:57Z https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/bc-p/143011#M872 <P>Databricks added the ability to parameterize the catalog and schema for Dashboards, I tested it and wrote detailed examples -&nbsp;<A href="https://medium.com/@protmaks/dynamic-catalog-schema-in-databricks-dashboards-b7eea62270c6" target="_self">https://medium.com/@protmaks/dynamic-catalog-schema-in-databricks-dashboards-b7eea62270c6</A></P><DIV><DIV class="">&nbsp;</DIV></DIV> Mon, 05 Jan 2026 10:57:34 GMT https://community.databricks.com/t5/technical-blog/customizing-target-deployments-in-databricks-asset-bundles/bc-p/143011#M872 protmaks 2026-01-05T10:57:34Z