I was having a similar issue in using .rdd.map()
Solved it by adding two key value pairs in the spark config for the cluster

spark.databricks.pyspark.enablePy4JSecurity false

spark.databricks.pyspark.trustedFilesystems org.apache.spark.api.java.JavaRDD

After this I was able to read the schema of the json from the column that was read as string