GIT automate DEV Databricks instance to PROD instance

[adrianhernandez]

Hello,
Would like to use GIT to automate the process of syncing between a DEV Databricks instance and a PROD Databricks instance. Something like :

1. On GIT console pull changes/sync with DEV Databricks.
2. Have some kind of approval process in GIT (like someone entering comments and then approving/submitting changes, only admins should do this so it should be role authentication based).
3. Once admin approves it the changes are pushed to PROD Databricks.

Any guidance, recommendations, how-tos, etc. are welcomed.

[szymon_dybczak]

Hi @adrianhernandez ,

This can be easily achieved using Databricks Assets Bundles in combination with Azure Devops Pipeline (or github actions).

So, typical CI/CD workflows looks something like this:

1. Store: Store your Databricks code and notebooks in a version control system like Git. 
   
   
2. Code: Develop code and unit tests in a Databricks notebook in the workspace or locally using an external IDE.
   
   
3. Build: Use Databricks Asset Bundles settings to automatically build certain artifacts during deployments. 
   
   
4. Deploy: Deploy changes to the Databricks workspace using Databricks Asset Bundles in conjunction with tools like Azure DevOps or GitHub Actions.

So, you can create flow that will be triggered when you merge your PR to your branch.When you deploy to higher environments like QA or PRD Azure Devops (and github actions) gives you ability to set up approval process. Possibilites are unlimited here 😄

Here you can find some pipelines using DABs for reference and inspiration:

GitHub - datakickstart/datakickstart_dabs: Example of project using Databricks Asset Bundle

Demo Deploy a Databricks Asset Bundle with Azure D... - Databricks Community - 126585

[adrianhernandez]

Do you know any examples similar but for AWS? So far all that I find in forums or searching is for Azure, and require Azure specific addons/artifacts to be used. Thanks.

[szymon_dybczak]

Yes, you can follow official guide on AWS documentation. They're using GitHub actions. 

And when you type in databricks assets bundles + GitHub actions you will find plenty of blogs/sites with examples.

https://docs.databricks.com/aws/en/dev-tools/ci-cd/github