Hi everyone,
I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deployment
What happened:
- I started a job that depends on a wheel file built by the bundle (timestamped artifact in .bundle/.../artifacts/.internal/)
- While the job was running, I ran databricks bundle deploy again
- The deployment generated a new timestamped wheel file and removed the old one
- My running job failed with ERROR_NO_SUCH_FILE_OR_DIRECTORY because it couldn't find the original artifact
My concern: This seems like it could be a problem in team environments. If two developers are working on the same bundle target:
- Developer A starts a job from their deployment
- Developer B deploys their changes to the same target
- Developer A's running job fails due to missing artifacts
My questions:
- Is this expected behavior, or am I misusing bundles?
- What are the recommended patterns to prevent this in multi-developer teams?
- Should each developer use personal bundle targets (dev_alice, dev_bob), or is there a better approach?
- Does this same issue apply to production deployments? If so, how should we handle long-running jobs during deployment?
- How should production CI/CD deployments be coordinated when scheduled or long-running jobs might be active? Should we check for active runs before deploying? Is there a built-in mechanism or recommended pattern to prevent breaking currently executing production jobs?
Any guidance on best practices for coordinating bundle deployments with active job runs would be appreciated!
