topic Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs in Data Engineering

"Databricks Bundle Deploy -t prod" command deletes log of historical runs

ynskrbn — Mon, 16 Dec 2024 13:24:30 GMT

I'm using Databricks Asset Bundles with Azure DevOps CI/CD for workflow deployment. While the initial deployment to production works fine, I encounter an issue when updating the workflow in the development environment and redeploying it to production. Upon redeployment, all historical run logs are lost. Is there a way to preserve these logs during deployment, perhaps using a command like databricks bundle reset -t prod?

It should display all the log like this

but after deploying everything is gone

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

Alberto_Umana — Mon, 16 Dec 2024 14:04:35 GMT

Hi @ynskrbn,

This is likely because the deployment process overwrites the existing job definitions, which in turn resets the job history. To preserve historical logs, you need to ensure that the job definitions are not completely overwritten during the redeployment process. Instead, you should update the existing jobs while retaining their history

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

ynskrbn — Mon, 16 Dec 2024 14:09:34 GMT

I know how to do this with the API, but not with Databricks Asset Bundles in my deployment pipeline. where can i find the documentation?

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

Alberto_Umana — Mon, 16 Dec 2024 14:37:14 GMT

Please let me know if this documentation helps you: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

PabloCSD — Tue, 17 Dec 2024 19:42:19 GMT

When you re-deploy you job, do you augment the version? (e.g., 4.3.0 -> 4.3.1)

I have been through this, when I change a definition in the databricks.yml, for example when changing the bundle name, because it detects as a new workflow.

Can you explain about the "re-deployment"? In our Azure DevOps CI/CD pipeline, when we generate a new release of the workflow it is just used this command:

databricks bundle deploy --target prod

topic Re: &quot;Databricks Bundle Deploy -t prod&quot; command deletes log of historical runs in Data Engineering

"Databricks Bundle Deploy -t prod" command deletes log of historical runs

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs

topic Re: "Databricks Bundle Deploy -t prod" command deletes log of historical runs in Data Engineering