Exclude a job from bundle deployment in PROD

[ChingizK]

My question is regarding Databricks Asset Bundles. I have defined a databricks.yml file the following way:

bundle:
  name: my_bundle_name

include:
  - resources/jobs/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: workspace_url

  prod:
    mode: production
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: service_principal_name

In the example above I have several individual job YML files defined in the /resources/jobs/ directory. So the folder contains v1_job.yml, v2_job.yml & v3_job.yml files. When I execute databricks bundle deploy -t prod command all the 3 jobs get deployed. Is there a way to exclude some of the jobs in this case? For example, I only need v3 job to be deployed in prod, even though I need all 3 in dev

[giuseppegrieco]

Hello, if you want, you can deploy specific jobs only in the development environment. Since you have only two environments, a straightforward approach is to modify your jobs YAML definition as follows:

resources:
  jobs:
    # Define the jobs to be deployed in all environments here

targets:
  dev:
    resources:
      jobs:
        # Define your development-only jobs here

[Isa1]

I have all my job YAML definitions in separate files and I add them using 'include'. Is there a way to then exclude a job from a specific target without having to add the whole YAML definition to the databricks.yml file?

[hiryucodes]

I'm also looking for this use case. Have you found any solution? There really should be an "exclude" option at target level.

[Pelle123]

I also need this.

[Coffee77]

Me too. No clean solution yet. As workaround I implemented first an "extra" control in specific jobs that never should be run in PROD to block execution based on environment variable in all clusters (I don't really like much but it was effective). As a second workaround, only updating pipelines in a customized way to not include jobs from the beginning as explaining first in this topic or deleting them upon being deployed in order to not make your folder code structure too much complex, I don't really like much either xD, but what to do instead?