topic Excluding job update from DAB .yml deployment in Data Engineering

Excluding job update from DAB .yml deployment

jar — Fri, 29 Aug 2025 03:12:14 GMT

Hi.

We have a range of scheduled jobs and _one_ continuous job all defined in .yml and deployed with DAB. The continuous job is paused per default and we use a scheduled job of a notebook to pause and unpause it so that it only runs during business hours. The issue is that if we want to deploy changes to existing scheduled jobs or a new job, it'll override the status defined by the notebook with the paused status defined in the .yml file.

It seems it's possible to define which jobs to deploy with the "include" key in the databricks.yml file, but as we have _many_ scheduled jobs and _one_ continuous job, this seems a rather circumstantial solution both to implement but also to maintain. Isn't it possible to negate the include definition, which would be a lot more elegant? Is there another solution I am not aware of?

Thanks in advance for any contribution.

Re: Excluding job update from DAB .yml deployment

Yogesh_Verma_ — Fri, 29 Aug 2025 03:39:52 GMT

You’re running into this because DAB treats the YAML definition as the source of truth — so every time you redeploy, it will reset the job state (including the paused/running status) back to what’s defined in the file. Unfortunately, there isn’t currently a way to negate include so that you can easily exclude just the continuous job from deployments.

The common workarounds are:

Split the continuous job into a separate bundle so it’s managed independently of the scheduled jobs.

Use include explicitly for the jobs you want to update, even though it requires maintaining the list.

Or, manage the continuous job outside of DAB if its state needs to be controlled dynamically (e.g. via notebook or API).

At the moment, DAB doesn’t support excluding resources by default, so either separating resources into different bundles or explicitly including what you want are the only reliable options.