Hubert-Dudek
Databricks MVP

It would be best if you used Databricks ML runtime (in cluster settings), not the standard one.


My blog: https://databrickster.medium.com/

View solution in original post

yes, exactly, this is what I have figured it out yesterday, and it wasted so many of my hours troubleshooting it. But at least it is solved, thanks.

Hubert's suggestion is the solution. Thanks.

done.