topic DAB - Add/remove task depending on workspace. in Machine Learning

DAB - Add/remove task depending on workspace.

p4pratikjain — Mon, 19 Aug 2024 14:53:54 GMT

I use DAB for deploying Jobs, I want to add a specific Task in dev only but not in staging or prod. Is there any way to achieve this using DAB ?

Re: DAB - Add/remove task depending on workspace.

mark_ott — Thu, 13 Nov 2025 11:36:35 GMT

Yes, you can add a task in Databricks Asset Bundles (DAB) that exists only in the development environment and not in staging or production. This is handled by defining environment-specific resources using the "targets" mapping in your bundle configuration. For example, in your bundle.yml or equivalent configuration file, you specify tasks or jobs under each target (such as dev, staging, prod). Only tasks or jobs defined within the dev target will be deployed to that environment, and will not appear in other targets like staging or prod.

To do this:

Define your dev-only task under the resources section inside the dev target in your bundle configuration.
Do not include that task under the staging or prod targets.

This modular structure makes it easy to have environment-specific jobs or tasks, such as dev-only test or experiment tasks, without affecting staging or production deployments.

Re: DAB - Add/remove task depending on workspace.

Coffee77 — Thu, 13 Nov 2025 12:21:01 GMT

You can define specific resources by target in DAB as shown here. This is valid for jobs and/or tasks:

For instance, in my case:

I think, best option (but not available as far as I know) would be to be able to define "include" sections by target, instead of having to put all resources code @mark_ott

Re: DAB - Add/remove task depending on workspace.

Pat — Thu, 05 Feb 2026 11:55:55 GMT

I know it's a bit old, but if someone is looking into a solution, then I was able to resolve the issue where I need to deploy some jobs only into the DEV target:

https://github.com/databricks/bundle-examples/tree/main/knowledge_base/target_includes.

Use YAML anchors to define resources once and include them only in specific targets.

File:resources/test_jobs.yml

# Define anchor with job configuration job-config: &job-config my_test_job: name: "Test Job" tasks: - task_key: test_task notebook_task: notebook_path: tests/my_test.py # Reference anchor only in dev target targets: dev: resources: jobs: <<: *job-config # Merge anchor here uat: # Empty - no test jobs in uat prd: # Empty - no test jobs in prd

File: databricks.yml

bundle:
  name: my_bundle

include:
  - ./resources/test_*.yml  # Include test jobs file

targets:
  dev:
    # test_jobs.yml adds resources here
  uat:
    # No test jobs
  prd:
    # No test jobs

Result:

dev target gets my_test_job
uat and prd targets have no test jobs