dbfs not found

[byrnesy5]

Hi, 

I've saved a custom pyfunc and now I'm trying to load it in a pandas_udf. It works on small samples or if I repartition everything to 1 partition, but when I try to run it on a larger sample and distribute it across my cluster it fails repeatably and says the model in the dbfs cannot be found. 

Any ideas?

Thanks,

Andrew

[byrnesy5]

Here you can see the progress. 15 tasks succeeded and 110 have failed (eventually the job will fail completely).  Seems like some workers can see the file and others cannot. Not sure why some would fail and others would succeed in this instance. 

[Alberto_Umana]

This problem can often be attributed to the model artifacts not being available on all the executors, especially in a distributed environment.

Can you try using the dbutils.fs.refreshMounts() in your code?

If the model is small enough, broadcast it to all executors using sc.broadcast

[byrnesy5]

Thanks for getting back to me. Didn't have any luck though. 

I tried dbutils.fs.refreshMounts() and still getting the same errors. 

I tried broadcasting the model, but it's not able to be pickled and broadcast.

Any other ideas?