shan_chandra
Databricks Employee
Databricks Employee

@chaitanya​ , could you please try disabling arrow optimization and see if this resolves the issue?

spark.sql.execution.arrow.enabled false

spark.sql.execution.arrow.pyspark.enabled false

View solution in original post