Automating Version Control for Databricks Workflows

[Prasad329]

I am currently using Databricks Asset Bundles to manage and deploy workflows. While I have successfully automated the version control for notebooks, I am facing challenges with workflows. Specifically, I am looking to automate the process of fetching changes made in the Databricks UI back to the GitHub repository.

Current Workflow:

1. I create and manage workflows in the Databricks UI.
2. To update the GitHub repository, I manually export the workflows YAML file from the UI and commit it to the GitHub repository.
3. I then use GitHub Actions with Databricks CLI bundle commands to redeploy the updated workflows as a bundle.

After deploying workflows as bundle, any changes made directly in the Databricks UI are not automatically fetched and updated in the GitHub repository. The UI shows a notification: "This task was deployed as part of a bundle, avoid editing this copy, instead edit the source and redeploy the bundle." This requires manual intervention to disconnect the workflow from the bundle, make changes, export the YAML, and commit it to the repository.

Requirement: I need to automate the process such that any changes made in the Databricks workflows from the UI are automatically fetched, committed to the GitHub repository, and redeployed as bundles without manual intervention.

Any insights or suggestions would be greatly appreciated!

[-werners-]

using the UI that is not possible I think.
When using DAB and YML files it can be done.
So I suggest you create the workflow using the UI (because it is easy to use) and then create a DAB out of that (using bundle generate).
I admit, there is still some work to asset bundles.