cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

darthdickhead
by New Contributor III
  • 8680 Views
  • 4 replies
  • 3 kudos

Best way to install and manage a private Python package that has a continuously updating Wheel

I'm trying to setup a Workspace Library that is used internally within our organization. This is a Python package, where the source is available on a private GitHub repository, and not accessible on PyPi or the wider internet / surface web. I managed...

  • 8680 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Eshwaran Venkat​ You can use the Databricks CLI to automate the process of cloning the private GitHub repository and building/uploading the Python package to DBFS as a wheel file. You can schedule this process to run periodically, such as once a day...

  • 3 kudos
3 More Replies
tatekeller
by New Contributor
  • 1684 Views
  • 1 replies
  • 0 kudos

Can you access a repo file in an init script?

I'd like to configure a cluster with python libraries as defined in a requirements file. I have a pip requirements.txt file in a private repo which I have integrated on Databricks (and I can access it through the UI and view it on Databricks). I upda...

  • 1684 Views
  • 1 replies
  • 0 kudos
Latest Reply
sher
Valued Contributor II
  • 0 kudos

you can install in a cluster

  • 0 kudos
Labels