cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error using score_batch for batch inference

Ariane
New Contributor II

Hey everybody,

I have been learning to use the Databricks feature store and I was trying to train the model using the stored features and compute batch inference. 

I am getting an error though, running prediction using score_batch, I have been getting this error for the past two days and I have tried everything its still not resolving.  I have checked my input batch dataframe(spark_df), and there are no null values, no infinite values, and no values that are beyond the float 32. Something am I missing?

Ariane_0-1692892706534.png

 

3 REPLIES 3

Kumaran
Databricks Employee
Databricks Employee

Hi @Ariane,

Thank you for posting your question in the Databricks community.

If you are using the RandomForestRegressorCould you check by setting maxDepth it to the default value to mitigate this issue?

Ariane
New Contributor II

Hey @Kumaran

I am using a Random forest classifier however I have tried to set the max depth to none since it is the default value but the error still exists. 

mariabardon
New Contributor II

Has this issue been resolved? I am using a Wrapped Catboost classifier, and sometimes I can use batch_score, and some times it gives me the same error:

"org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 2094.0 failed 4 times, most recent failure: Lost task 2.3 in stage 2094.0 (TID 2825) (10.162.80.108 executor 0): org.apache.spark.SparkRuntimeException: [UDF_ERROR.ENV_LOST] Execution of function udf(named_struct(..."

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group