I was having a similar issue in using .rdd.map()Solved it by adding two key value pairs in the spark config for the clusterspark.databricks.pyspark.enablePy4JSecurity falsespark.databricks.pyspark.trustedFilesystems org.apache.spark.api.java.JavaRDD ...