Hi Team, one of my customer is facing the below issue.. Anyone faced this issue before ? Any help would be appreciated.
import mlflow
mlflow.set_registry_uri("databricks-uc")
catalog_name = "system"
embed = mlflow.pyfunc.spark_udf(spark, f"models:/system.ai.bge_m3/1", "array<float>")
On running the above piece of code, we are getting the below error
TypeError: ColSpec.__init__() got an unexpected keyword argument 'required'
WARNING mlflow.pyfunc: Detected one or more mismatches between the model's dependencies and the current Python environment: - mlflow (current: 2.7.1, required: mlflow==2.11.2) - torch (current: 2.0.1+cu118, required: torch==2.2.1) - transformers (current: 4.31.0, required: transformers==4.38.2) To fix the mismatches, call `mlflow.pyfunc.get_model_dependencies(model_uri)` to fetch the model's environment and install dependencies using the resulting environment file.
WARNING mlflow.pyfunc: Calling `spark_udf()` with `env_manager="local"` does not recreate the same environment that was used during training, which may lead to errors or inaccurate predictions. We recommend specifying `env_manager="conda"`, which automatically recreates the environment that was used to train the model and performs inference in the recreated environment.
