Bundle deployment overwrites artifacts while jobs are running - best practices?

[animeshjain]

Hi everyone,

I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deployment

What happened:

1. I started a job that depends on a wheel file built by the bundle (timestamped artifact in .bundle/.../artifacts/.internal/)
2. While the job was running, I ran databricks bundle deploy again
3. The deployment generated a new timestamped wheel file and removed the old one
4. My running job failed with ERROR_NO_SUCH_FILE_OR_DIRECTORY because it couldn't find the original artifact

My concern: This seems like it could be a problem in team environments. If two developers are working on the same bundle target:

• Developer A starts a job from their deployment
• Developer B deploys their changes to the same target
• Developer A's running job fails due to missing artifacts

My questions:

1. Is this expected behavior, or am I misusing bundles?
2. What are the recommended patterns to prevent this in multi-developer teams?
3. Should each developer use personal bundle targets (dev_alice, dev_bob), or is there a better approach?
4. Does this same issue apply to production deployments? If so, how should we handle long-running jobs during deployment?
5. How should production CI/CD deployments be coordinated when scheduled or long-running jobs might be active? Should we check for active runs before deploying? Is there a built-in mechanism or recommended pattern to prevent breaking currently executing production jobs?

Any guidance on best practices for coordinating bundle deployments with active job runs would be appreciated!

[sudhaktr]

Do you have source_linked_deployment set as false? That's probably causing it.

[animeshjain]

no that not what I am talking about its like this in the picture we can generate a build artifact and use this in the job
so if a developer run the deploy and running there job and the same time 2nd deploy happens which overwrites the build. so now the first job is looking for whl which is overwritten

[sudhaktr]

Yes I understand that part. if you have source_linked_deployment set as false, both the developers will be deploying to the same location under /.bundle directory. Then the overwrite can happen.

If source_linked_deployment is set as True or not set(by default it is True), then the workflow will be pointing to the source. That is respective developer's directory.