@ck7007 

Your suggestion to use pandas works. Thank you! For others my tests results:

On separate clusters 16.4 LTS and scala 2.13 a I have tested the same code and have different results:

Cluster 1) PySparkTypeError: Exception thrown when converting pandas.Series (object) with name 'column1' to Arrow Array (None).

Cluster 2) Works fine with warnings

On cluster with runtime 17.1 works without warinings.

THIS CODE DOES NOT WORK using DATABRICKS CONNECT (when you develop from IDE).

@szymon_dybczak  thank you for your check.