Since last year, we have adopted Databricks Asset Bundles for deploying our workflows to the production and staging environments. The tool has proven to be quite effective, and we currently use Azure DevOps Pipelines to automate bundle deployment, triggered by pull requests and changes to our repository.
However, we have observed unexpected behavior: on two occasions, including one instance two weeks ago, our jobs deployed in production were duplicated, without any changes to our deployment routine. It is important to note that this issue manifested exclusively in the production environment, while the staging workspace remains stable and without duplications, despite both using the same deployment process, varying only the target ('uat' or 'prod') in the pipeline's YAML file.
While searching for solutions, I found reports from other Databricks community users who have experienced a similar issue, as discussed in this thread: https://community.databricks.com/t5/data-engineering/azure-pipeline-databricks-bundle-deploy-duplica...
Although the discussion does not provide a definitive solution or an official acknowledgment of the issue by Databricks, it reinforces the possibility that we might be facing a genuine, existing issue.
So, here are a few points I've noticed:
Based on my analysis, I suspect the problem might be related to the Databricks Asset Bundle's state management. The bundle maintains state files that record important information about the deployment, such as the job name and ID, and the path to the YAML file used for the deployment.
Normally, during a standard deployment, these files are updated with the new information if there are changes to the job or when a new job is added. However, during the duplication incidents, we noticed that the 'Created at' column of the files is updated to the date the issue occurred. This suggests that, instead of simply being updated, the state files are being recreated during these events. This recreation results in the assignment of new job IDs in the YAML file, leading to the creation of a new job with the same name but with a different ID, causing the duplication.
The only way we can resolve this is by manually changing these state files back to the previous job IDs. After performing a new bundle deployment, the system returns to normal operation without duplication.
CLI version: 0.245
