topic Deploy tar.gz package from private git hub in Data Engineering

Deploy tar.gz package from private git hub

thushar — Wed, 19 Jan 2022 11:07:37 GMT

We created Python package (.tar.gz) and kept it under private git.

We can able to connect to that git (using PAT) from the Azure databricks notebook.

Our requirement is to install that package from .tar.gz file for that notebook

"pip install https://USERNAME:PASWWORD@github.com/company_github/my_repo/my_package.tar.gz";

getting the below error.

does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

Re: Deploy tar.gz package from private git hub

Anonymous — Wed, 19 Jan 2022 16:32:24 GMT

Hello @Thushar R - My name is Piper, and I'm a moderator for Databricks. It's great to have you here and thank you for your question. 🙂

Let's give the community a while to answer before we circle back around to this.

Re: Deploy tar.gz package from private git hub

Rahul_Samant — Thu, 20 Jan 2022 10:23:29 GMT

For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/

alternatively you can pass the tar.gz using --py-files while submitting job.

Re: Deploy tar.gz package from private git hub

thushar — Fri, 21 Jan 2022 08:12:55 GMT

Thanks, but why do I need to build again the package?

I am using VS Code for development, through VS Code I built my package and generated tar.gz and wheel file and committed in Git. So the aim is to install the already built package from Git to Azure databricks notebook.

What I understood from your reply is, even the package exists, we have to build it again for installing, so we need the setup.py file, right?

Re: Deploy tar.gz package from private git hub

Atanu — Sat, 12 Feb 2022 16:18:48 GMT

If i understood correctly , yes that's correct you need to use the py file. is Rahul's solution worked @Thushar R ?