Getting python version errors when using pyspark rdd using databricks connect

[Surajv]

Hi community, 

When I use pyspark rdd related functions in my environment using databricks connect, I get below error: 

Databricks cluster version: 12.2. 

`RuntimeError: Python in worker has different version 3.9 than that in driver 3.10, PySpark cannot run with different minor versions. Please check environment variables PYSPARK PYTHON...`

How can I resolve it? 

[Kaniz]

Hi @Surajv, The error message you’re encountering indicates a Python version mismatch between the Spark worker and the Spark driver.

To resolve this issue, follow these steps:

1. Install Correct Python Version on Worker Node:

   • Ensure that the correct Python version (Python 3) is installed on the worker node.
   • Add Python 3 to the system’s PATH.
   • Set the PYSPARK_PYTHON environment variable to "python3" on the worker node.
   • To verify, run pyspark in the terminal on the worker node. This will open a Python shell, and you can check the Python version displayed at the top of the shel...¹.

2. Update Spark Configuration:

   • Edit or create the ./conf/spark-defaults.conf file (you can copy it from spark-defaults.conf.template) on both the master and worker nodes.
   • Add the following line to the configuration file:

     spark.pyspark.python /usr/bin/python3
     

   • Save the changes and restart both the master and worker nodes.

3. Verify Environment Variables:

   • Ensure that the environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
   • These variables should point to the same Python version (Python 3) used by your packages and Spark

[Surajv]

Got it. As a side note, I tried above methods, but the error persisted, hence upon reading docs again, there was this statement: You must install Python 3 on your development machine, and the minor version of your client Python installation must be the same as the minor Python version of your Databricks cluster. (Link: https://docs.databricks.com/en/dev-tools/databricks-connect-legacy.html#requirements). 

On aligning my environment python version with Databricks cluster python version, the error got resolved.