topic Error using score_batch for batch inference in Machine Learning

Error using score_batch for batch inference

Ariane — Thu, 24 Aug 2023 16:00:32 GMT

Hey everybody,

I have been learning to use the Databricks feature store and I was trying to train the model using the stored features and compute batch inference.

I am getting an error though, running prediction using score_batch, I have been getting this error for the past two days and I have tried everything its still not resolving. I have checked my input batch dataframe(spark_df), and there are no null values, no infinite values, and no values that are beyond the float 32. Something am I missing?

Re: Error using score_batch for batch inference

Kumaran — Thu, 24 Aug 2023 17:20:07 GMT

Hi @Ariane,

Thank you for posting your question in the Databricks community.

If you are using the RandomForestRegressor, Could you check by setting maxDepth it to the default value to mitigate this issue?

Re: Error using score_batch for batch inference

Ariane — Mon, 28 Aug 2023 09:58:57 GMT

Hey @Kumaran,

I am using a Random forest classifier however I have tried to set the max depth to none since it is the default value but the error still exists.

Re: Error using score_batch for batch inference

mariabardon — Fri, 23 Aug 2024 07:53:56 GMT

Has this issue been resolved? I am using a Wrapped Catboost classifier, and sometimes I can use batch_score, and some times it gives me the same error:

"org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 2094.0 failed 4 times, most recent failure: Lost task 2.3 in stage 2094.0 (TID 2825) (10.162.80.108 executor 0): org.apache.spark.SparkRuntimeException: [UDF_ERROR.ENV_LOST] Execution of function udf(named_struct(..."