topic Databricks Asset Bundle - Not fully deploying in Azure Pipeline in Data Engineering

Databricks Asset Bundle - Not fully deploying in Azure Pipeline

KuruDev — Wed, 21 Aug 2024 20:24:48 GMT

Hello Community,

I'm encountering a challenging issue with my Azure Pipeline and I'm hoping someone here might have some insights. I'm attempting to deploy a Databricks bundle that includes both notebooks and workflow YAML files. When deploying the bundle locally via the Databricks CLI, everything works perfectly – both notebooks and YAML files are uploaded correctly into my databricks environment.

Also I identified, that if I run "databricks bundle deploy" on my local environment, it creates three folders in databricks:

- artifacts, files, state

But if I deploy on my azure pipeline the "state"-folder is also missing.

The problem arises when I try to deploy the same bundle through my Azure Pipeline. In this case, only the notebooks are being uploaded, and the workflow YAML files are missing. I've made sure that the CLI versions match and that the pipeline configuration is correct. The verbose logs don't show any obvious errors, and the environment variables all seem to be in order as well. It just does not deploy my workflows.

Has anyone experienced a similar issue or have any idea what might be going wrong here? Any tips or directions to investigate would be greatly appreciated.

Thank you in advance for your support!

Best Regards

Re: Databricks Asset Bundle - Not fully deploying in Azure Pipeline

KuruDev — Wed, 21 Aug 2024 20:25:42 GMT

Not the workflow yaml files are missing, i meant, that it doesnt create my workflows.

Re: Databricks Asset Bundle - Not fully deploying in Azure Pipeline

jtberman — Wed, 20 Nov 2024 02:16:41 GMT

Were you ever able to find a resolution for the issue? I am facing the same thing where it deploys the files fine but doesn't actually create the workflow.

Re: Databricks Asset Bundle - Not fully deploying in Azure Pipeline

adfo — Wed, 20 Nov 2024 09:17:51 GMT

Hello,

Same issue here, files and wheel are deployed and present on the databricks workspace but the jobs are not created