Hi.

The problem might be with pickling a language model.

Have you tried to use mlflow.spark.log_model to save the model? Spark ML models cannot be serialized as pickle files. They are serialized in a language-neutral hierarchical format that can be read by both Python and Scala as in the "sparkml" directory below.

spark-model
+-sparkml/
| +-stages/
| | +-1_DecisionTreeRegressor_6aae1e6c3fed/
| | | +-data/
| | | | +-part-00000-a4b9cb99-abd2-40c3-90d2-a46b44926263-c000.snappy.parquet
| | | | +-.part-00000-a4b9cb99-abd2-40c3-90d2-a46b44926263-c000.snappy.parquet.crc
| | | | +-._SUCCESS.crc
| | | |
| | | +-metadata/
| | |   +-part-00000
| | |   +-.part-00000.crc
| | |   +-._SUCCESS.crc
| | |  
| | +-0_VectorAssembler_ce8bcea8c5b3/
| |   +-metadata/
| |     +-part-00000
| |     +-.part-00000.crc
| |     +-._SUCCESS.crc
| |    
| +-metadata/
|   +-part-00000
|   +-.part-00000.crc
|   +-._SUCCESS.crc
|  
+-requirements.txt
+-python_env.yaml
+-conda.yaml
+-MLmodel

Another resource on Models an deployment is this post on Medium "Effortless models deployment with Mlflow — Packing a NLP product review classifier from HuggingFace" (https://santiagof.medium.com/effortless-models-deployment-with-mlflow-packing-a-nlp-product-review-classifier-from-huggingface-13be2650333)