Databricks asset bundle occasionally duplicating jobs

gsouza — Wed, 26 Mar 2025 20:40:39 GMT

Since last year, we have adopted Databricks Asset Bundles for deploying our workflows to the production and staging environments. The tool has proven to be quite effective, and we currently use Azure DevOps Pipelines to automate bundle deployment, triggered by pull requests and changes to our repository.

However, we have observed unexpected behavior: on two occasions, including one instance two weeks ago, our jobs deployed in production were duplicated, without any changes to our deployment routine. It is important to note that this issue manifested exclusively in the production environment, while the staging workspace remains stable and without duplications, despite both using the same deployment process, varying only the target ('uat' or 'prod') in the pipeline's YAML file.

While searching for solutions, I found reports from other Databricks community users who have experienced a similar issue, as discussed in this thread: https://community.databricks.com/t5/data-engineering/azure-pipeline-databricks-bundle-deploy-duplicating-jobs/td-p/82591.

Although the discussion does not provide a definitive solution or an official acknowledgment of the issue by Databricks, it reinforces the possibility that we might be facing a genuine, existing issue.

So, here are a few points I've noticed:

Based on my analysis, I suspect the problem might be related to the Databricks Asset Bundle's state management. The bundle maintains state files that record important information about the deployment, such as the job name and ID, and the path to the YAML file used for the deployment.

Normally, during a standard deployment, these files are updated with the new information if there are changes to the job or when a new job is added. However, during the duplication incidents, we noticed that the 'Created at' column of the files is updated to the date the issue occurred. This suggests that, instead of simply being updated, the state files are being recreated during these events. This recreation results in the assignment of new job IDs in the YAML file, leading to the creation of a new job with the same name but with a different ID, causing the duplication.

The only way we can resolve this is by manually changing these state files back to the previous job IDs. After performing a new bundle deployment, the system returns to normal operation without duplication.

CLI version: 0.245

Re: Databricks asset bundle occasionally duplicating jobs

NandiniN — Thu, 01 May 2025 06:20:01 GMT

Hi,

There have been internal discussions on this issue in the past and this issue is reported. There are internal trackers for this issue.

Thanks!

Re: Databricks asset bundle occasionally duplicating jobs

isabelgontijo — Mon, 05 May 2025 12:53:57 GMT

Hi @NandiniN

Thank you for your reply, but we would like to know if there is any way for us to monitor the progress of the solution to this problem. It is a recurring error and it is very disruptive when it occurs.

Best regards.

Re: Databricks asset bundle occasionally duplicating jobs

kevin_w_edwards — Mon, 02 Jun 2025 20:00:14 GMT

This is negatively impacting us as well.

Re: Databricks asset bundle occasionally duplicating jobs

cmantilla — Wed, 10 Dec 2025 15:21:36 GMT

This is a recurring issue for my org as well.

topic Databricks asset bundle occasionally duplicating jobs in Data Engineering

Databricks asset bundle occasionally duplicating jobs

Re: Databricks asset bundle occasionally duplicating jobs

Re: Databricks asset bundle occasionally duplicating jobs

Re: Databricks asset bundle occasionally duplicating jobs

Re: Databricks asset bundle occasionally duplicating jobs