This problem can often be attributed to the model artifacts not being available on all the executors, especially in a distributed environment.

Can you try using the dbutils.fs.refreshMounts() in your code?

If the model is small enough, broadcast it to all executors using sc.broadcast