Error using score_batch for batch inference

Ariane · ‎08-24-2023

Hey everybody,

I have been learning to use the Databricks feature store and I was trying to train the model using the stored features and compute batch inference.

I am getting an error though, running prediction using score_batch, I have been getting this error for the past two days and I have tried everything its still not resolving. I have checked my input batch dataframe(spark_df), and there are no null values, no infinite values, and no values that are beyond the float 32. Something am I missing?

Kumaran · ‎08-24-2023

Hi @Ariane,

Thank you for posting your question in the Databricks community.

If you are using the RandomForestRegressor, Could you check by setting maxDepth it to the default value to mitigate this issue?

Ariane · ‎08-28-2023

Hey @Kumaran,

I am using a Random forest classifier however I have tried to set the max depth to none since it is the default value but the error still exists.

mariabardon · ‎08-23-2024

Has this issue been resolved? I am using a Wrapped Catboost classifier, and sometimes I can use batch_score, and some times it gives me the same error:

"org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 2094.0 failed 4 times, most recent failure: Lost task 2.3 in stage 2094.0 (TID 2825) (10.162.80.108 executor 0): org.apache.spark.SparkRuntimeException: [UDF_ERROR.ENV_LOST] Execution of function udf(named_struct(..."