cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Getting python version errors when using pyspark rdd using databricks connect

Surajv
New Contributor III

Hi community, 

When I use pyspark rdd related functions in my environment using databricks connect, I get below error: 

Databricks cluster version: 12.2. 

`RuntimeError: Python in worker has different version 3.9 than that in driver 3.10, PySpark cannot run with different minor versions. Please check environment variables PYSPARK PYTHON...`

How can I resolve it? 

1 REPLY 1

Surajv
New Contributor III

Got it. As a side note, I tried above methods, but the error persisted, hence upon reading docs again, there was this statement: You must install Python 3 on your development machine, and the minor version of your client Python installation must be the same as the minor Python version of your Databricks cluster. (Link: https://docs.databricks.com/en/dev-tools/databricks-connect-legacy.html#requirements). 

On aligning my environment python version with Databricks cluster python version, the error got resolved. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group