Anonymous
Not applicable

@Eshwaran Venkat​ 

You can use the Databricks CLI to automate the process of cloning the private GitHub repository and building/uploading the Python package to DBFS as a wheel file. You can schedule this process to run periodically, such as once a day, using a cron job or a similar scheduling mechanism.

  1. Install and configure the Databricks CLI on your local machine or a separate server.
  2. Create a Python script that clones the private GitHub repository, builds the Python package, and uploads it to DBFS as a wheel file. You can use the git command and the setuptools package to perform these tasks.
  3. Add the script to a cron job or a similar scheduling mechanism to run it periodically, such as once a day.
  4. In your Databricks notebooks, install the Python package from the uploaded wheel file in DBFS.

This approach allows you to synchronize the private GitHub repository and install the Python package in Databricks with minimal manual intervention.