Be aware, that in Databricks 15.2 LTS this behavior is broken.
I cannot find the code, but most likely related to the following option:
https://github.com/apache/spark/commit/c1c710e7da75b989f4d14e84e85f336bc10920e0#diff-f9ddcc6cba651c6...

I was able to reproduce the issue locally when the latest pyarrow is installed, with this option enabled.