Hello Kaniz

I am currently using:

• pyspark: 3.5.0 which is default in Spark ML 14.3LTS runtime
• xgboost : 1.7.6

I have also checked the driver logs and there seems to be no problems because of some UDF(S). Anything else that can be tried?

I checked the code works perfectly fine with a single node cluster but somehow throws this error Multinode cluster. Here are the configurations of two clusters:

1. Single Node cluster

• Data Bricks runtime version: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)
• Node Type: Standard_D4ds_v5

The code runs perfectly fine in this one.

2. Multi Node Interactive cluster

• Data Bricks runtime version: 14.3 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)
• Node Type: Standard_D4ds_v5
• Min Workers: 1
• Max Workers 3
  
  

How is this happening that two clusters with same runtime and library version but one runs perfectly fine but other throws this error?