cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is it possible to clone a private repository and use it in databricks Repos?

Andyfcx
New Contributor

As title, I need to clone code from my private git repo, and use it in my notebook,

I do something like

def cmd(command, cwd=None):
   process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, cwd=cwd)
   output, error = process.communicate()
   print (output.decode())
   if error is not None:
      print ("----\n"+error.decode())

But I have to do this, redefine this function all the time.

I think this is cumbersome,

Is there a way that I could clone the repo into the "Repos" which databricks offered in the left side bar, and have full version control with databricks?

(I am not installing published packages from pip)

1 ACCEPTED SOLUTION

Accepted Solutions

Prabakar
Databricks Employee
Databricks Employee

Hi @Andy Huang​ , Yes, you can do it if it's accessible from Databricks. Please refer to:

https://docs.databricks.com/repos.html#repos-for-git-integration

Databricks does not support private Git servers, such as Git servers behind a VPN.

View solution in original post

2 REPLIES 2

Prabakar
Databricks Employee
Databricks Employee

Hi @Andy Huang​ , Yes, you can do it if it's accessible from Databricks. Please refer to:

https://docs.databricks.com/repos.html#repos-for-git-integration

Databricks does not support private Git servers, such as Git servers behind a VPN.

Hubert-Dudek
Esteemed Contributor III

I think it does support private git servers just for example in Azure you need to set network routing - vnet injection ( https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/vn... ) and whitelist your internet gateway on git. So it will be seemed as any other git.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group