I used the script provided by databricks as I use the autoML incorporated in databricks, so I would not assume that it modifies the ML runtime. I think there must have been something wrong with the cluster configuration, because I have created a new cluster and now it seems to work.