topic Re: Bundle deployment overwrites artifacts while jobs are running - best practices? in Data Engineering

Bundle deployment overwrites artifacts while jobs are running - best practices?

animeshjain — Sat, 27 Jun 2026 11:51:17 GMT

Hi everyone,

I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deployment

What happened:

I started a job that depends on a wheel file built by the bundle (timestamped artifact in .bundle/.../artifacts/.internal/)
While the job was running, I ran databricks bundle deploy again
The deployment generated a new timestamped wheel file and removed the old one
My running job failed with ERROR_NO_SUCH_FILE_OR_DIRECTORY because it couldn't find the original artifact

My concern: This seems like it could be a problem in team environments. If two developers are working on the same bundle target:

Developer A starts a job from their deployment
Developer B deploys their changes to the same target
Developer A's running job fails due to missing artifacts

My questions:

Is this expected behavior, or am I misusing bundles?
What are the recommended patterns to prevent this in multi-developer teams?
Should each developer use personal bundle targets (dev_alice, dev_bob), or is there a better approach?
Does this same issue apply to production deployments? If so, how should we handle long-running jobs during deployment?
How should production CI/CD deployments be coordinated when scheduled or long-running jobs might be active? Should we check for active runs before deploying? Is there a built-in mechanism or recommended pattern to prevent breaking currently executing production jobs?

Any guidance on best practices for coordinating bundle deployments with active job runs would be appreciated!

Re: Bundle deployment overwrites artifacts while jobs are running - best practices?

sudhaktr — Sat, 27 Jun 2026 14:09:58 GMT

Do you have source_linked_deployment set as false? That's probably causing it.

Re: Bundle deployment overwrites artifacts while jobs are running - best practices?

animeshjain — Sat, 27 Jun 2026 14:30:48 GMT

no that not what I am talking about its like this in the picture we can generate a build artifact and use this in the job
so if a developer run the deploy and running there job and the same time 2nd deploy happens which overwrites the build. so now the first job is looking for whl which is overwritten

Re: Bundle deployment overwrites artifacts while jobs are running - best practices?

sudhaktr — Sat, 27 Jun 2026 14:54:51 GMT

Yes I understand that part. if you have source_linked_deployment set as false, both the developers will be deploying to the same location under /.bundle directory. Then the overwrite can happen.

If source_linked_deployment is set as True or not set(by default it is True), then the workflow will be pointing to the source. That is respective developer's directory.