cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Excluding job update from DAB .yml deployment

jar
Contributor

Hi.

We have a range of scheduled jobs and _one_ continuous job all defined in .yml and deployed with DAB. The continuous job is paused per default and we use a scheduled job of a notebook to pause and unpause it so that it only runs during business hours. The issue is that if we want to deploy changes to existing scheduled jobs or a new job, it'll override the status defined by the notebook with the paused status defined in the .yml file. 

It seems it's possible to define which jobs to deploy with the "include" key in the databricks.yml file, but as we have _many_ scheduled jobs and _one_ continuous job, this seems a rather circumstantial solution both to implement but also to maintain. Isn't it possible to negate the include definition, which would be a lot more elegant? Is there another solution I am not aware of?

Thanks in advance for any contribution.

1 REPLY 1

Yogesh_378691
Contributor

You’re running into this because DAB treats the YAML definition as the source of truth — so every time you redeploy, it will reset the job state (including the paused/running status) back to what’s defined in the file. Unfortunately, there isn’t currently a way to negate include so that you can easily exclude just the continuous job from deployments.

The common workarounds are:

Split the continuous job into a separate bundle so it’s managed independently of the scheduled jobs.

Use include explicitly for the jobs you want to update, even though it requires maintaining the list.

Or, manage the continuous job outside of DAB if its state needs to be controlled dynamically (e.g. via notebook or API).

At the moment, DAB doesn’t support excluding resources by default, so either separating resources into different bundles or explicitly including what you want are the only reliable options.

Yogesh Verma

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now