topic Re: Databricks Asset Bundle Workflow Redeployment Issue in Data Engineering

Databricks Asset Bundle Workflow Redeployment Issue

Sikki — Sat, 27 Apr 2024 05:55:44 GMT

Hello All,

In my Databricks workflows, I have three tasks configured, with the final task set to run only if the condition "ALL_DONE" is met. During the first deployment, I observed that the dependency "ALL_DONE" was correctly assigned to the last task. However, upon subsequent redeployments, I noticed that "ALL_DONE" was also being assigned to other tasks. Upon further investigation, it appears that the naming of the task keys may be causing this discrepancy. This hypothesis was confirmed when I tested with different task key names, such as task-A, task-B, and task-C, and found that the workflow configuration was correctly maintained during redeployment as well.
Thanks,Sikki

Re: Databricks Asset Bundle Workflow Redeployment Issue

Sikki — Sat, 27 Apr 2024 06:10:20 GMT

Hi, I've attached two snapshots, one from the initial deployment and another from the second redeployment, where no changes were made to the code.

Re: Databricks Asset Bundle Workflow Redeployment Issue

Yeshwanth — Mon, 29 Apr 2024 06:24:37 GMT

Hi @Sikki Good day!

There was an issue and it was fixed recently. Could you please confirm if you are still facing the issue?

Best regards,

Re: Databricks Asset Bundle Workflow Redeployment Issue

Sikki — Mon, 29 Apr 2024 09:26:47 GMT

The issue hasn't been resolved yet. Should I update my Databricks CLI?

Re: Databricks Asset Bundle Workflow Redeployment Issue

Sikki — Mon, 29 Apr 2024 10:14:59 GMT

The issue hasn't been resolved yet. Should I update my Databricks CLI to generate new bundle_config_schema.json.

Re: Databricks Asset Bundle Workflow Redeployment Issue

Yeshwanth — Mon, 29 Apr 2024 10:17:21 GMT

@Sikki are you using the latest version of the CLI? v.28?

Re: Databricks Asset Bundle Workflow Redeployment Issue

Sikki — Mon, 29 Apr 2024 17:53:25 GMT

Hi @Yeshwanth I am using latest CLI version and using Azure DevOps to deploy the workflow.

Re: Databricks Asset Bundle Workflow Redeployment Issue

Sikki — Tue, 30 Apr 2024 15:42:37 GMT

After updating my CLI, I successfully deployed the job from Databricks CLI and it is functioning correctly. However, when attempting to deploy the same job using Azure DevOps, I encounter the same issue.

Re: Databricks Asset Bundle Workflow Redeployment Issue

Yeshwanth — Wed, 01 May 2024 06:17:18 GMT

@Sikki thanks for confirming.

For Azure DevOps please check the version of Databricks CLI installed.