It looks like there are other issues. I saved the model generated with the code above in mlflow
When I try to reload it with this code:
import mlflow
model = mlflow.spark.load_model('runs:/cb6ff62587a0404cabeadd47e4c9408a/model')
It works in a notebook, but it failed when running it in intelliJ with databricks-connect. I get this error:
mlflow.exceptions.MlflowException: Failed to download an "MLmodel" model file from "runs:/cb6ff62587a0404cabeadd47e4c9408a/model"
Do you have any idea where it could come from?
*Note that I do not have this problem with a sklearn model