We trained a Spark ML FMRegressor model and registered it to Unity Catalog via MLflow. When attempting to load it back using mlflow.spark.load_model, we get an

OSError: [Errno 5] Input/output error: '/dbfs/tmp' regardless of what dfs_tmpdir path is passed.

Tried:

Using mlflow.pyfunc.spark_udf as an alternative also fails — when the features VectorUDT column is serialized through pandas during UDF execution, it loses its type and becomes a plain StructType, causing an IllegalArgumentException at inference time.

Does anyone have a fix for this?