topic Re: Having trouble installing my own Python wheel? in Data Engineering

Having trouble installing my own Python wheel?

397973 — Fri, 05 Apr 2024 14:52:48 GMT

I want to install my own Python wheel package on a cluster but can't get it working. I tried two ways:

I followed these steps: https://docs.databricks.com/en/workflows/jobs/how-to/use-python-wheels-in-workflows.html#:~:text=March%2025%2C%202024,code%20in%20your%20Databricks%20jobs.

but it failed on #8 of Step 5, "In the Add dependent library dialog, with Workspace selected, drag the my_test_package-0.0.1-py3-none-any.whl file created in step 4 into the dialog’s Drop file here area"

l created the wheel in the previous steps but the Workspace dialog doesn't have "Drop file here". I also uploaded the wheel to Workspace but it doesn't allow me to select it.

I also tried this: https://docs.databricks.com/en/libraries/notebooks-python-libraries.html

In the step:

%pip install /Workspace/Users/myname@mycompany.com/my_test_package-0.0.1-py3-none-any.whl

It said

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/Workspace/Users/myname@mycompany.com/my_test_package-0.0.1-py3-none-any.whl'

even though I see the wheel there.

Is there another way? What I have is pretty simple, just custom functions. I was hoping this would be easier.

Re: Having trouble installing my own Python wheel?

shan_chandra — Fri, 05 Apr 2024 17:44:17 GMT

@397973 - Once you uploaded the .whl file, did you had a chance to list the file manually in the notebook?

Also, did you had a chance to move the files to /Volumes .whl file?

Re: Having trouble installing my own Python wheel?

397973 — Fri, 05 Apr 2024 18:40:37 GMT

Actually, no I can't even find it. I see it in the browser Workspace, but when I do "%ls" it shows

azure/

eventlogs/

logs/

conf/

hadoop_accessed_config.lst*

preload_class.lst*

Re: Having trouble installing my own Python wheel?

shan_chandra — Fri, 05 Apr 2024 18:57:29 GMT

@397973 - can you please copy the relative file path from the UI and use them as a dependent library in your cluster?