I am a member of the development team in our company and we use Databricks as sort of like ETL tool. We utilize git integration for our program and run Workflow daily basis. Recently, we created another company internal private git repository and want to automatically install/use these packages in our program. I know we can install private repository by specifying repository URL with authentication information in requirements.txt as follows.
ใปgit+https://<your-username>:<your-token>@github.com/<your-username>/<your-private-repo>.git@main#egg=<package-name>
ใปgit+ssh://git@github.com/<your-username>/<your-private-repo>.git@main#egg=<package-name>
However, these methods are the ways tied to the individual, so I suppose it's not suitable for large project. ใ
I found there is this token called "Deploy Token" which does not tie to individual account, rather links to specific repository, but apparently you need to store secret key into cluster every time you run Workflow.
Is there any ways to install/use packages in private repository for another program, or is there any missing ideas/features that I don't know about Databricks?
