DAB - Add/remove task depending on workspace.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-19-2024 07:53 AM
I use DAB for deploying Jobs, I want to add a specific Task in dev only but not in staging or prod. Is there any way to achieve this using DAB ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2025 03:36 AM
Yes, you can add a task in Databricks Asset Bundles (DAB) that exists only in the development environment and not in staging or production. This is handled by defining environment-specific resources using the "targets" mapping in your bundle configuration. For example, in your bundle.yml or equivalent configuration file, you specify tasks or jobs under each target (such as dev, staging, prod). Only tasks or jobs defined within the dev target will be deployed to that environment, and will not appear in other targets like staging or prod.
To do this:
-
Define your dev-only task under the
resourcessection inside thedevtarget in your bundle configuration. -
Do not include that task under the
stagingorprodtargets.
This modular structure makes it easy to have environment-specific jobs or tasks, such as dev-only test or experiment tasks, without affecting staging or production deployments.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-13-2025 04:17 AM - edited 11-13-2025 04:21 AM
You can define specific resources by target in DAB as shown here. This is valid for jobs and/or tasks:
For instance, in my case:
I think, best option (but not available as far as I know) would be to be able to define "include" sections by target, instead of having to put all resources code @mark_ott
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-05-2026 03:55 AM
I know it's a bit old, but if someone is looking into a solution, then I was able to resolve the issue where I need to deploy some jobs only into the DEV target:
Use YAML anchors to define resources once and include them only in specific targets.
File:resources/test_jobs.yml
# Define anchor with job configuration
job-config: &job-config
my_test_job:
name: "Test Job"
tasks:
- task_key: test_task
notebook_task:
notebook_path: tests/my_test.py
# Reference anchor only in dev target
targets:
dev:
resources:
jobs:
<<: *job-config # Merge anchor here
uat:
# Empty - no test jobs in uat
prd:
# Empty - no test jobs in prdFile: databricks.yml
bundle: name: my_bundle include: - ./resources/test_*.yml # Include test jobs file targets: dev: # test_jobs.yml adds resources here uat: # No test jobs prd: # No test jobs
Result:
- dev target gets my_test_job
- uat and prd targets have no test jobs