Re: Facing issues databricks asset bundle, All jo...

mark_ott · ‎10-31-2025

The issue you’re facing—where all Databricks Asset Bundle jobs are being deployed to all targets instead of only the defined target(s)—appears to be a known limitation in how the bundle resource inclusion and target mapping works in the Databricks CLI, especially with YAML structure from templates like default-python and when using top-level resource includes.

Why This Happens

When you use the include: directive in your databricks.yaml, all resource files matched by the glob pattern (e.g., resources/*.yml) are merged into every target in the bundle.
This means every resource (job, cluster, etc.) that is part of the bundle’s include expression ends up being available in every target, unless you explicitly restructure your YAML to scope resources by target.
As a result, databricks bundle deploy --target dev (or any target) will deploy all jobs included, rather than limiting to just the ones you want.
There is currently no official CLI flag or YAML key like deployments: [target] that allows you to restrict the deployment of individual assets/jobs to specific targets only—every included resource ends up in every target automatically.

Workarounds and Best Practices

1. Move Resources Under Target Blocks

The most robust solution is to avoid only using global includes and, instead, define your jobs or resources within the appropriate targets: section. Only resources defined within a target will be deployed for that target.

Example:

text

targets:
  dev:
    resources:
      jobs:
        asset_bundles_job1:
          # job1 definition here
        asset_bundles_job2:
          # job2 definition here
  dev_ca:
    resources:
      jobs:
        asset_bundles_job3:
          # job3 definition here

This way, only job1 and job2 deploy to dev, only job3 deploys to dev_ca.

2. Use Separate Bundles for Different Environments

If jobs differ substantially between environments, you can maintain multiple bundle YAMLs (one per environment), or as sub-bundles, and avoid broad includes. This can be scripted at build/deploy time.

3. Manually Refactor Resource Includes

Instead of a glob pattern like include: resources/*.yml, explicitly include only the resource YAMLs you want for each target. This is less maintainable at scale, but is effective for small setups.

4. Monitor the Databricks CLI GitHub Issues

There is an open issue discussing adding support to exclude global resources for certain targets. Keep track of this for future enhancements.

Current Limitation

As of Databricks CLI v0.240.0 and the documented June/October 2025 CLI behavior, resource inclusion is always global unless restructured as above.

In summary:

Move target-specific jobs under each target inside databricks.yaml, or use separate includes for each target instead of a global pattern.
There is no native way to limit deployment of individual jobs to some targets unless jobs are housed within those targets’ blocks in the configuration.
Watch for updates addressing exclusion/inclusion per target, as this remains a feature request.

For your use case, restructuring your databricks.yaml so that each target defines only the jobs it should receive is the recommended workaround for now.