cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Custom transformers with mlflow

NaeemS
New Contributor III

Hi Everyone,

I have created a spark pipeline in which I have a stage which is a Custom Transformer. Now I am using feature stores to log my model. But the issue is that the custom Transformer stage is not serialized properly and is not logged along with the whole pipeline. I have logged the pipeline using MLFLOW along with logging the custom transformer as an artifact. Then in inference environment I loaded the model's artifact first and saved it in a temporary path and added that path to the sys paths making that custom code available for the model to do inference successfully. But I can not do this when I log my model using feature stores. Also, this method of logging the custom code as an artifact and loading it before the inference is not reliable and kills the overall idea of end to end pipeline. 

Any help in this regard would be highly appreciated.

Thanks in advance!

 

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @NaeemS

  • Custom Transformers in Spark ML pipelines are not directly serializable by default. The Spark ML framework provides serialization for built-in transformers, but custom ones require additional handling.
  • This serialization limitation is likely to be the cause of the issue you’re facing with feature stores not properly logging the custom Transformer stage.

NaeemS
New Contributor III

Hi @Kaniz_Fatma , Can you please guide me what are the additional steps I'll need to handle serialization of Custom transformers so I can use it in my model pipeline via feature stores.

Thanks!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!