topic Re: Exclude a job from bundle deployment in PROD in Data Engineering

Exclude a job from bundle deployment in PROD

ChingizK — Mon, 24 Jun 2024 17:45:13 GMT

My question is regarding Databricks Asset Bundles. I have defined a databricks.yml file the following way:

bundle: name: my_bundle_name include: - resources/jobs/*.yml targets: dev: mode: development default: true workspace: host: workspace_url prod: mode: production workspace: root_path: /Shared/.bundle/${bundle.name}/${bundle.target} run_as: service_principal_name: service_principal_name

In the example above I have several individual job YML files defined in the /resources/jobs/ directory. So the folder contains v1_job.yml, v2_job.yml & v3_job.yml files. When I execute databricks bundle deploy -t prod command all the 3 jobs get deployed. Is there a way to exclude some of the jobs in this case? For example, I only need v3 job to be deployed in prod, even though I need all 3 in dev

Re: Exclude a job from bundle deployment in PROD

giuseppegrieco — Mon, 24 Jun 2024 18:50:33 GMT

Hello, if you want, you can deploy specific jobs only in the development environment. Since you have only two environments, a straightforward approach is to modify your jobs YAML definition as follows:

resources: jobs: # Define the jobs to be deployed in all environments here targets: dev: resources: jobs: # Define your development-only jobs here

Re: Exclude a job from bundle deployment in PROD

Isa1 — Mon, 24 Feb 2025 10:12:36 GMT

I have all my job YAML definitions in separate files and I add them using 'include'. Is there a way to then exclude a job from a specific target without having to add the whole YAML definition to the databricks.yml file?

Re: Exclude a job from bundle deployment in PROD

hiryucodes — Tue, 11 Mar 2025 09:24:18 GMT

I'm also looking for this use case. Have you found any solution? There really should be an "exclude" option at target level.

Re: Exclude a job from bundle deployment in PROD

Pelle123 — Mon, 25 Aug 2025 12:38:41 GMT

I also need this.

Re: Exclude a job from bundle deployment in PROD

Coffee77 — Mon, 25 Aug 2025 13:02:39 GMT

Me too. No clean solution yet. As workaround I implemented first an "extra" control in specific jobs that never should be run in PROD to block execution based on environment variable in all clusters (I don't really like much but it was effective). As a second workaround, only updating pipelines in a customized way to not include jobs from the beginning as explaining first in this topic or deleting them upon being deployed in order to not make your folder code structure too much complex, I don't really like much either xD, but what to do instead?