ColumnTransformer not fitted after sklearn Pipeline loaded from Mlflow

[Nasreddin]

I am building a machine learning model using sklearn Pipeline which includes a ColumnTransformer as a preprocessor before the actual model. Below is the code how the pipeline is created.

transformers = []
num_pipe = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])
transformers.append(('numerical', num_pipe, num_cols))
cat_pipe = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('ohe', OneHotEncoder(handle_unknown='ignore'))
])
transformers.append(('categorical', cat_pipe, cat_cols))
preprocessor = ColumnTransformer(transformers, remainder='passthrough')
model = Pipeline([
  ('prep', preprocessor),
  ('clf', XGBClassifier())
])

I am using Mlflow to log the model artifact as sklearn model after it is fitted on training data.

model.fit(X, y)
mlflow.sklearn.log_model(model, model_uri)

When I tried to load the model from mlflow for scoring though, I got the error "This ColumnTransformer instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator."

run_model = mlflow.sklearn.load_model(model_uri)
run_model.predict(X_pred)

I also ran check_is_fitted on the second step of the Pipeline which is the xgboost model itself after loaded from mlflow and it is NOT fitted either.

Is Mlflow not compatible with sklearn Pipeline with multiple steps?

[Kaniz_Fatma]

Hi @ Nasreddin! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Kaniz_Fatma]

Hi @Nasreddin, MLflow is compatible with sklearn Pipeline with multiple steps. The error you're encountering, "This ColumnTransformer instance is not fitted yet. Call’ fit’ with appropriate arguments before using this estimator." is likely because  ColumnTransformer of the Pipeline is not fitted correctly before being logged to MLflow. Looking at your code, it seems you are fitting and logging the model accurately. The issue might be with the loading of the model. When you load the model from MLflow, ensure that the model_uri correctly points to the logged model. Also, provide the model was successfully logged and saved in the correct path. Please verify these points and try again. If the problem persists, it might be a good idea to check the versions of sklearn, xgboost, and MLflow you are using and ensure they are compatible. 

Sources:
1. [Docs: models-in-uc-example](https://docs.databricks.com/mlflow/models-in-uc-example.html)
2. [Docs: workspace-model-registry-example](https://docs.databricks.com/mlflow/workspace-model-registry-example.html)
3. [Docs: scikit-learn](https://docs.databricks.com/machine-learning/preprocess-data/scikit-learn.html)