Hey @Łukasz 

Thanks for reporting.

As I see Spark 3.4.0 introduced an improvement that looks to be the cause for this issue.

Improvement: https://issues.apache.org/jira/browse/SPARK-37099

Similar Bug: https://issues.apache.org/jira/browse/SPARK-44448

This improvement [SPARK-37099] is included as part of DBR 13.1: https://docs.databricks.com/release-notes/runtime/13.1.html

That is the reason you are seeing this in DBR 13.1

As I have verified internally this seems to be fixed in DBR 13.1. I would request you to test it again once and let us know.