cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to refer the ".whl" files from repos into a Databricks "python wheel" task in Databricks jobs?

Maverick1
Valued Contributor II

I was trying to see whether it would be possible to refer a library from a Repos path (as “files in repos” feature is already available).

I can see the “.whl” file in the repo itself but it is not visible under the “Workspace” option of “python wheel” task in the DB jobs.

PFB the snaps:

“.whl” file is visible in Repo:

Screenshot 2022-03-21 at 7.20.51 PM 

“.whl” file not available for installation while defining the library path for wheel file in DB jobs task.

Screenshot 2022-03-21 at 7.21.25 PM 

1 ACCEPTED SOLUTION

Accepted Solutions

Maverick1
Valued Contributor II

As of now, The only resolutions for this query are:

  1. Either push/deploy the library to an DBFS or S3 path.
  2. Either deploy the library on a repos path and then refer it inside a notebook via passing repos path.

Accessing Workspace path or root "file:/" path is not possible.

View solution in original post

4 REPLIES 4

hotrabattecom
New Contributor II

I also have the same problem as you right now. I try to discover it but I dont know where to find it. Btw here is my site for shopping, https://hotrabatte.com/ .If you want, figure it out.

Mr__E
Contributor II

We set up a GitHub workflow to upload the built python wheels to a shared S3 cluster, which is accessible from the workspaces page, then we can add using the dialogue you show. However, I'm having an issue using it for job clusters, so YMMV.

Maverick1
Valued Contributor II

@Erik Louie​ : S3 and DBFS path are anyway accessible for job and all-purpose cluster. The query is that there is a "WORKSPACE" option available but still you can't access a library placed on workspace/repos path.

Maverick1
Valued Contributor II

As of now, The only resolutions for this query are:

  1. Either push/deploy the library to an DBFS or S3 path.
  2. Either deploy the library on a repos path and then refer it inside a notebook via passing repos path.

Accessing Workspace path or root "file:/" path is not possible.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group