was this resolved?

I was having a similar issue in using .rdd.map()
Solved it by adding two key value pairs in the spark config for the cluster

spark.databricks.pyspark.enablePy4JSecurity false

spark.databricks.pyspark.trustedFilesystems org.apache.spark.api.java.JavaRDD

After this I was able to read the schema of the json from the column that was read as string 

Did you tried this in a UC enabled cluster?

In my case the problem was that we were trying to use SparkXGBoostRegressor and in the docs it says that it does not work on clusters with autoscaling enabled. So we just disabled autoscaling for the interactive cluster where we were testing the model and it worked like a charm 🙂

Hope it helps

Hello,
In the past I used 

rdd.mapPartitions(lambda ...)

 to call functions that access third party APIs like azure ai translate text to batch call the API and return the batched data.

How would one do this now? 

Hi All,

I faced the same issue. When i updated the cluster to DBR 15.5 it got resolved, it is not working in DBR 13.3 (my previous cluster version).

Well, it is also working in DBR 10.4. I stayed with 15.5 as being the latest. Just try out with different DBR version and it should solve your issue.