Faced this issue multiple times.

Solution:

1. Don't use Shared Cluster or cluster without Unity Catalog enabled for running 'rdd' queries on Databricks.

2. Instead create a Personal Cluster (Single User) with basic configuration and with Unity Catalog enabled.

3. Also for the new compute cluster in Advanced Options set the following parameters:

1. Under Spark Config:
   • spark.databricks.driver.disableScalaOutput true
   • spark.databricks.delta.preview.enabled true
2. Under Environment Variables:
   • PYSPARK_PYTHON=/databricks/python3/bin/python3

Re-run your rdd queries with new compute cluster. It works perfectly well for me.