DimitriBlyumin
New Contributor III

Looks like it is a known issue/limitation due to Parquet internals, and it will not be fixed. Apparently there is no workaround in Spark.

https://issues.apache.org/jira/browse/SPARK-27442