Hello @lingareddy_Alva,

Thank you very much for your answer. I feel like the spark.sql.execution.arrow.useLargeVarTypes config not being respected by the transformWithStateInPandas serializer path is a bug. Is there somewhere I could report it to the Spark development team? I am unfortunatly not a Scala or Java dev and cannot send a patch myself. If I could, I would...

Regarding your suggestions to work around the issue, we are already using the binary type for the file contents. Spliting the files in chunks is unfortunatly not very feasable for us. However, I don't even use the file's content in the stateful processor, I use other columns around it. The file's content is use in a later stage of our pipeline, so I really only need to have it pass throught the stateful processor to have it available on the other side. I was thinking that it might be possible to split my stream in 2 branches, one that sends the relevant columns to the stateful processor and the file content goes "around" it and the the 2 branches are stiched back together with a join. I am however worried about join 2 streaming datasets together, especially when there is statefulness involved.

Thank you again for your help. I will wait for a few days before marking your answer as accepted in case you have the time to write me back, because it really did answer my original question regarding the OversizedAllocationException exception.

Best regards

TWBDE