Hello,

I have a process that uses transformWithStateInPandas on a dataframe that has the content on entire files in on of the columns. Recently, the exception OversizedAllocationException has started happening. I have tried setting these configs in the job cluster's definitions, with no success.

spark.sql.execution.arrow.pyspark.enabled true
spark.sql.execution.arrow.transformWithStateInPandas.maxRecordsPerBatch 10
spark.sql.execution.arrow.useLargeVarTypes true
spark.sql.execution.pythonUDF.arrow.enabled true

I have also tried maxRecordsPerBatch at 1 and still got issues. And the message is always that is tried to allocate over 2GB. It's almost as if useLargeVarTypes is ignored for transformWithStateInPandas. I have also tried dissabling arrow with:

spark.sql.execution.arrow.pyspark.enabled false
spark.sql.execution.pythonUDF.arrow.enabled true

I am using Databricks runtime 17.3 LTS, so Spark 4.0.0. My code is python code with PySpark.

Thank you for any help you can give me.