topic Re: Deploy Workflow only to specific target (Databricks Asset Bundles) in Administration & Architecture

Deploy Workflow only to specific target (Databricks Asset Bundles)

johnb1 — Mon, 16 Sep 2024 09:49:32 GMT

I am using Databricks Asset Bundles to deploy Databricks workflows to all of my target environments (dev, staging, prod). However, I have one specific workflow that is supposed to be deployed only to the dev target environment.

How can I implement that this specific workflow is only deployed to dev target?

I was searching for information to do that on the workflow level (Create a new job | Jobs API | REST API reference | Azure Databricks). However, according this does not seem to be possible.

Any other location where I can specify this?

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

filipniziol — Mon, 16 Sep 2024 11:58:27 GMT

Hi @johnb1 ,
To deploy to specifitc target, you can deploy with -t flag:

databricks bundle deploy -t dev

Check the docs:
bundle command group | Databricks on AWS

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

szymon_dybczak — Mon, 16 Sep 2024 12:23:56 GMT

Hi @johnb1 ,

You can achieve what you want by using targets mapping. For dev target define job in resource submapping that you only want to have in that environment.

https://docs.databricks.com/en/dev-tools/bundles/settings.html

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

johnb1 — Mon, 16 Sep 2024 12:51:17 GMT

Thank you for that suggestion @szymon_dybczak. This looks helpful. My workflow is already defined in another .yaml file, as I have stored all workflow configs - each in one file - in a central folder. Thing is I do not want to define the workflow under targets in my DAB root file (mostly called databricks.yaml), as you suggested, but rather reference it. Is this possible?

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

szymon_dybczak — Mon, 16 Sep 2024 13:58:35 GMT

Hi @johnb1 ,

I'm not sure if you can reference yaml in targets mapping, but I can check it tommorrow morning.
But if I had to guess, based on the documentation entry below, I would say that there is currently no way to reference another yaml file in targets mapping:

"The include array specifies a list of path globs that contain configuration files to include within the bundle. These path globs are relative to the location of the bundle configuration file in which the path globs are specified.

The Databricks CLI does not include any configuration files by default within the bundle. You must use the include array to specify any and all configuration files to include within the bundle, other than the databricks.yml file itself.

This include array can appear only as a top-level mapping."

So, the easiest way to achieve what you want is to do something similiar. But, tommorrow I will confirm if we can reference yaml file.

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

johnb1 — Mon, 16 Sep 2024 13:57:26 GMT

Thanks @szymon_dybczak that would be great 👌

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

johnb1 — Fri, 20 Sep 2024 08:41:46 GMT

Hi @szymon_dybczak

Did you have a chance to check if such referencing of a workflow from another yaml file is possible?

Given you did not reply, I assume it is not possible. This would also confirm what I found during my tests.

If you know otherwise, however, please let me know 😀

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

szymon_dybczak — Fri, 20 Sep 2024 09:00:37 GMT

Hi @johnb1 ,

Sorry, I checked it but forgot to post reply. I can confirm that currently it's not possible. I'll setup a request for this feature on databricks idea portal.

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

johnb1 — Tue, 24 Sep 2024 08:43:10 GMT

Thanks for getting back and clarifying @szymon_dybczak

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

anishumesh — Tue, 04 Feb 2025 12:29:06 GMT

Any update to this solution yet? @szymon_dybczak

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

meghanav_hmc — Wed, 12 Feb 2025 18:28:09 GMT

Hi, I'm also looking to deploy different jobs in different targets. And these jobs are defined in a separate .yml file and we'll need to reference these jobs in the targets accordingly. Any updates on this implementation?

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

asham — Wed, 30 Apr 2025 13:40:27 GMT

Hi, also curious about this update, this would be a really helpful feature for us since we have humongous job specifications that are specific for dev and test environments. Adding these to the top level databricks.yml file is really cluttering up our config. I've tried overriding the resources on prod with:

job: job_name: {}

and variations of this, hoping this would override the job with an empty list, but it doesn't seem to work since it just doesn't override anything.

Re: Deploy Workflow only to specific target (Databricks Asset Bundles)

Pat — Thu, 05 Feb 2026 11:56:58 GMT

I know it's a bit old, but if someone is looking into a solution, then I was able to resolve the issue where I need to deploy some jobs only into the DEV target:

https://github.com/databricks/bundle-examples/tree/main/knowledge_base/target_includes.

Use YAML anchors to define resources once and include them only in specific targets.

File:resources/test_jobs.yml

# Define anchor with job configuration job-config: &job-config my_test_job: name: "Test Job" tasks: - task_key: test_task notebook_task: notebook_path: tests/my_test.py # Reference anchor only in dev target targets: dev: resources: jobs: <<: *job-config # Merge anchor here uat: # Empty - no test jobs in uat prd: # Empty - no test jobs in prd

File: databricks.yml

bundle:
  name: my_bundle

include:
  - ./resources/test_*.yml  # Include test jobs file

targets:
  dev:
    # test_jobs.yml adds resources here
  uat:
    # No test jobs
  prd:
    # No test jobs

Result:

dev target gets my_test_job
uat and prd targets have no test jobs