cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Automating Version Control for Databricks Workflows

Prasad329
New Contributor

I am currently using Databricks Asset Bundles to manage and deploy workflows. While I have successfully automated the version control for notebooks, I am facing challenges with workflows. Specifically, I am looking to automate the process of fetching changes made in the Databricks UI back to the GitHub repository.

Current Workflow:

  1. I create and manage workflows in the Databricks UI.
  2. To update the GitHub repository, I manually export the workflows YAML file from the UI and commit it to the GitHub repository.
  3. I then use GitHub Actions with Databricks CLI bundle commands to redeploy the updated workflows as a bundle.

After deploying workflows as bundle, any changes made directly in the Databricks UI are not automatically fetched and updated in the GitHub repository. The UI shows a notification: "This task was deployed as part of a bundle, avoid editing this copy, instead edit the source and redeploy the bundle." This requires manual intervention to disconnect the workflow from the bundle, make changes, export the YAML, and commit it to the repository.

Requirement: I need to automate the process such that any changes made in the Databricks workflows from the UI are automatically fetched, committed to the GitHub repository, and redeployed as bundles without manual intervention.

Any insights or suggestions would be greatly appreciated!

1 REPLY 1

-werners-
Esteemed Contributor III

using the UI that is not possible I think.
When using DAB and YML files it can be done.
So I suggest you create the workflow using the UI (because it is easy to use) and then create a DAB out of that (using bundle generate).
I admit, there is still some work to asset bundles.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group