topic Re: dbfs not found in Machine Learning

dbfs not found

byrnesy5 — Tue, 17 Dec 2024 16:21:03 GMT

Hi,

I've saved a custom pyfunc and now I'm trying to load it in a pandas_udf. It works on small samples or if I repartition everything to 1 partition, but when I try to run it on a larger sample and distribute it across my cluster it fails repeatably and says the model in the dbfs cannot be found.

Any ideas?

Thanks,

Andrew

Re: dbfs not found

byrnesy5 — Tue, 17 Dec 2024 16:43:18 GMT

Here you can see the progress. 15 tasks succeeded and 110 have failed (eventually the job will fail completely). Seems like some workers can see the file and others cannot. Not sure why some would fail and others would succeed in this instance.

Re: dbfs not found

Alberto_Umana — Tue, 17 Dec 2024 16:47:16 GMT

This problem can often be attributed to the model artifacts not being available on all the executors, especially in a distributed environment.

Can you try using the dbutils.fs.refreshMounts() in your code?

If the model is small enough, broadcast it to all executors using sc.broadcast

Re: dbfs not found

byrnesy5 — Tue, 17 Dec 2024 17:58:08 GMT

Thanks for getting back to me. Didn't have any luck though.

I tried dbutils.fs.refreshMounts() and still getting the same errors.

I tried broadcasting the model, but it's not able to be pickled and broadcast.

Any other ideas?