Hi,

Thanks for reaching out to Databricks forum,

This is a bug with OSS, which is being fixed in Spark 3 version.

Here is the jira ticket about the issue

https://issues.apache.org/jira/browse/SPARK-26208

Here is the pull request for the fix, which will be merged

https://github.com/apache/spark/pull/23173

Porting the fix to the Databricks runtime versions is in the pipeline.

Please let us know whether it answers your question or if you have follow-up question.

Thanks