Hi everyone,

We're experiencing an issue with Declarative Automation Bundles (DAB) while deploying to our Azure Databricks Production workspace through an Azure DevOps pipeline.

Environment

• Azure Databricks

• Declarative Automation Bundles (DAB)

• Azure DevOps Pipeline

• Authentication using a Service Principal

• Deployment target: prod

• Deployment lock enabled

Issue

During deployment, the Azure DevOps pipeline fails midway after successfully deploying some resources. As a result:

• A subset of jobs gets created successfully in the Databricks workspace.

• The deployment remains in a locked state.

• To recover, we rerun the deployment using:

databricks bundle deploy -t prod --force-lock

However, instead of updating the existing jobs that were already deployed, Databricks creates new duplicate jobs with different Job IDs.

Expected Behavior

I would expect the subsequent deployment to:

• Reuse the existing deployment state.

• Update the already deployed jobs.

• Continue deploying the remaining resources.

• Avoid creating duplicate jobs.

Current Behavior

Each retry with --force-lock creates duplicate jobs rather than updating the existing ones.

Questions

1. Is this expected behavior when using --force-lock after a partially successful deployment?

2. Is there a recommended recovery process after a deployment fails midway?

3. Is there a way to resume deployment without creating duplicate resources?

4. Does this indicate that the deployment state (Terraform/DAB state) is being recreated or lost?

5. Is there any recommended approach for Azure DevOps pipelines to prevent this scenario?

Any guidance or best practices would be greatly appreciated.

Thank you!