topic Re: Is it possible to clone a private repository and use it in databricks Repos? in Data Engineering

Is it possible to clone a private repository and use it in databricks Repos?

Andyfcx — Fri, 15 Oct 2021 10:11:08 GMT

As title, I need to clone code from my private git repo, and use it in my notebook,

I do something like

def cmd(command, cwd=None):
   process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, cwd=cwd)
   output, error = process.communicate()
   print (output.decode())
   if error is not None:
      print ("----\n"+error.decode())

But I have to do this, redefine this function all the time.

I think this is cumbersome,

Is there a way that I could clone the repo into the "Repos" which databricks offered in the left side bar, and have full version control with databricks?

(I am not installing published packages from pip)

Re: Is it possible to clone a private repository and use it in databricks Repos?

Prabakar — Fri, 15 Oct 2021 10:15:57 GMT

Hi @Andy Huang , Yes, you can do it if it's accessible from Databricks. Please refer to:

https://docs.databricks.com/repos.html#repos-for-git-integration

Databricks does not support private Git servers, such as Git servers behind a VPN.

Re: Is it possible to clone a private repository and use it in databricks Repos?

Hubert-Dudek — Fri, 15 Oct 2021 13:59:29 GMT

I think it does support private git servers just for example in Azure you need to set network routing - vnet injection ( https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/vnet-inject ) and whitelist your internet gateway on git. So it will be seemed as any other git.