cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

CI/CD pipeline using Github

Rsa
New Contributor II

Hi Team,
I've recently begun working with Databricks and I'm exploring options for setting up a CI/CD pipeline to pull the latest code from GitHub.
I have to pull latest code(.sql) from Github whenever push is done to main branch and update .sql notebook in Databricks. On scheduled run latest code should get executed.

I would greatly appreciate guidance on how to accomplish this. Thank you

@-werners-  @hubert_dudek@daniel_sahal  @Ajay-Pandey  @Rishabh-Pandey   @Aviral_Bhardwaj   @Vivian_Wilfred  @Pat  @karthik_p 

4 REPLIES 4

btafur
Databricks Employee
Databricks Employee

There are multiple alternatives for CI/CD on Databricks for deployment.

Rsa
New Contributor II

Hi @btafur ,

I went through the links. I found that we have to use 3rd service to pull the repo from Github to Databricks production folder like Jenkins or Github Action. For that we need to get separate license. Is it possible to pull code directly from Databricks?
Thanks for your help 

@daniel_sahal 

btafur
Databricks Employee
Databricks Employee

Generally you would require some sort of compute that executes the automation, so that's why using a third party tool, even if it is Open Source might incur in a small additional cost. However, some of those have free tiers as well depending on the tool. 

If all you need is pulling the code to Databricks, you can do it manually using Repos as mentioned in Option 1. However, any automation will require a server that runs the automation with any of the third party or Open Source tools - Jenkins, Github Actions, Terraform, etc.

-werners-
Esteemed Contributor III

FWIW:
we pull manually, but it is possible to automate that without any cost if you use Azure Devops.  There is a free tier (depending on the number of pipelines/duration).

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group