cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Uploading wheel using `dbutils.fs.cp` to workspace and install it in Runtime>15

safoineext
New Contributor

I have been trying to find an alternative to copying a wheel file from my local file system to Databricks and then installing it into the cluster. Doing this 

databricks_client.dbutils.fs.cp("file:/local..../..whl", "dbfs:/Workspace/users/..../..whl") used to work without issue for runtime 14 and below but not for 15 which I need for a project.
The other thing I don't get is that the file I copy to Databricks into the workspace using the Python SDK do not show up in the UI
safoineext_0-1720009993682.png

 

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @safoineext

  • Instead of copying wheel files directly to the cluster, consider using workspace libraries. These allow you to create an object in the Databricks UI to represent your library. The actual binary (the wheel file) will still reside on DBFS or PyPi, but you can select it from the cluster.
  • Another option is to upload your wheel files to Unity Catalog volumes. These volumes are shared across all clusters in your workspace.
    • By placing your wheel files in a Unity Catalog volume, you can easily reference them from any cluster without copying them directly.
    • This approach ensures consistency and avoids duplicating the wheel files for different clusters.
    • If you still prefer to copy wheel files directly to the cluster, use the following command:
      databricks fs cp local-path-to-wheel.whl dbfs:/mnt/catalog-volume/wheel-files/
      
    • This method allows you to upload the wheel file to DBFS and make it accessible to your clusters.
  • Databricks Repos also support arbitrary files, but using workspace libraries or Unity Catalog volumes is generally more straightforward for managing Python libraries in your workspace. 

Rishabh_Tiwari
Community Manager
Community Manager

Hi @safoineext ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,
Rishabh

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!