topic Re: Install custom wheel from dbfs in serverless enviroment in Data Engineering

Install custom wheel from dbfs in serverless enviroment

lezwon — Tue, 17 Jun 2025 05:30:00 GMT

Hey folks,

I have a job that runs on a serverless compute. I have also created a wheel file with custom functions, which I require in this job. I see that from here, we cannot install libraries for a task and must use notebook-scoped libraries. So what I am doing right now is:

1. Build the wheel locally
2. Push it to a location in dbfs (dbfs:/datalake/libs)
3. Install the library using %pip install /dbfs/datalake/libs/library.whl

But I get this error:
WARNING: Requirement '/dbfs/datalake/libs/library.whl' looks like a filename, but the file does not exist ERROR: library.whl is not a valid wheel filename.

I if do %fs ls dbfs:/datalake/libs/ i am able to see the wheel file.
But if i do !ls /dbfs/datalake/libs/, i do not see it.

So far, I'm having no luck installing the wheel file from dbfs. Can someone help me with some instructions or suggest a better way?

Re: Install custom wheel from dbfs in serverless enviroment

loui_wentzel — Wed, 18 Jun 2025 20:48:48 GMT

Is your dbfs mounted?

Otherwise, try uploading it to your workspace's "shared" folder - this is a common place to put these sorts of files. dbfs is slowly getting phased out and not really in any best practices.

Re: Install custom wheel from dbfs in serverless enviroment

lezwon — Thu, 19 Jun 2025 07:33:26 GMT

I created a new volume and copied the files in there. works well for me.