Uploading wheel using `dbutils.fs.cp` to workspace and install it in Runtime>15

[safoineext]

I have been trying to find an alternative to copying a wheel file from my local file system to Databricks and then installing it into the cluster. Doing this 

databricks_client.dbutils.fs.cp("file:/local..../..whl", "dbfs:/Workspace/users/..../..whl") used to work without issue for runtime 14 and below but not for 15 which I need for a project.
The other thing I don't get is that the file I copy to Databricks into the workspace using the Python SDK do not show up in the UI

[Kaniz_Fatma]

Hi @safoineext, 

• Instead of copying wheel files directly to the cluster, consider using workspace libraries. These allow you to create an object in the Databricks UI to represent your library. The actual binary (the wheel file) will still reside on DBFS or PyPi, but you can select it from the cluster.
  • Workspace libraries provide better visibility and management within the Databricks workspace.
  • Note that this approach doesn’t involve copying the wheel file directly to the cluster; it’s more about creating a reference to it.
  • You can find more information about workspace libraries in the Databricks documentation.
• Another option is to upload your wheel files to Unity Catalog volumes. These volumes are shared across all clusters in your workspace.
  • By placing your wheel files in a Unity Catalog volume, you can easily reference them from any cluster without copying them directly.
  • This approach ensures consistency and avoids duplicating the wheel files for different clusters.
  • If you still prefer to copy wheel files directly to the cluster, use the following command:

    databricks fs cp local-path-to-wheel.whl dbfs:/mnt/catalog-volume/wheel-files/
    

  • This method allows you to upload the wheel file to DBFS and make it accessible to your clusters.
• Databricks Repos also support arbitrary files, but using workspace libraries or Unity Catalog volumes is generally more straightforward for managing Python libraries in your workspace. 

[Rishabh_Tiwari]

Hi @safoineext ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,
Rishabh