Hi everyone,
I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deployment
What happened:
- I started a job that depends on a wheel file built by the bundle (timestamped artifact in .bundle/.../artifacts/.internal/)
- While the job was running, I ran databricks bundle deploy again
- The deployment generated a new timestamped wheel file and removed the old one
- My running job failed with ERROR_NO_SUCH_FILE_OR_DIRECTORY because it couldn't find the original artifact
My concern: This seems like it could be a problem in team environments. If two developers are working on the same bundle target:
- Developer A starts a job from their deployment
- Developer B deploys their changes to the same target
- Developer A's running job fails due to missing artifacts
My questions:
- Is this expected behavior, or am I misusing bundles?
- What are the recommended patterns to prevent this in multi-developer teams?
- Should each developer use personal bundle targets (dev_alice, dev_bob), or is there a better approach?
- Does this same issue apply to production deployments? If so, how should we handle long-running jobs during deployment?
- How should production CI/CD deployments be coordinated when scheduled or long-running jobs might be active? Should we check for active runs before deploying? Is there a built-in mechanism or recommended pattern to prevent breaking currently executing production jobs?
Any guidance on best practices for coordinating bundle deployments with active job runs would be appreciated!
no that not what I am talking about its like this in the picture we can generate a build artifact and use this in the job
so if a developer run the deploy and running there job and the same time 2nd deploy happens which overwrites the build. so now the first job is looking for whl which is overwritten
