Spark connect client and server versions should be same for executing UDFs

[chinmay0924]

I am trying to execute pandas UDF in databricks. It gives me the following error on serverless compute,

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b11ff17c-9b25-4ccb-927d-06a7d1ca7221/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:2340, in SparkConnectClient._handle_rpc_error(self, rpc_error) 2325 raise Exception( 2326 "Python versions in the Spark Connect client and server are different. " 2327 "To execute user-defined functions, client and server should have the " (...) 2336 "https://docs.databricks.com/en/release-notes/serverless.html" target="_blank" rel="noopener noreferrer">https://docs.databricks.com/en/release-notes/serverless.html</a>.</span><span class="ansi-yellow-fg">"

As far as I am aware, I cannot change the python version of the client in serverless. Are there any ways to fix this issue from my side?

[Louis_Frolio]

Correct, you cannot alter any pre-installed libraries on Serverless.  Have you considered changing your client side Python to align with the version running on the Serverless cluster?

[chinmay0924]

If I am using the databricks notebook with serverless attached, how do I change the client side python? I have control over client side python if I use databricks connect from my local setup, but is there a way to change the client side python version from databricks itself?

[Louis_Frolio]

When using a Databricks notebook attached to serverless compute, the Python version on the client side is determined by the underlying Serverless client image. This version is fixed and coupled with the Databricks Runtime (DBR) version or the Client Image version in the serverless environment. For example, Serverless Client Image v1 is tied to a specific Python version (e.g., Python 3.10.12 for DBR 14.3), and switching the DBR version changes the Python environment in use.

Currently, there is no direct way in Databricks to modify the default client-side Python version in serverless notebooks without changing the underlying DBR version or using custom measures like modifying deployment images.

For now, if you want to control the Python version for development purposes, using Databricks Connect locally remains your best option, as it allows full flexibility in configuring the desired Python environment on your local machine.

Hope this helps, Big Roux.

[chinmay0924]

Thanks a lot for the reply! Final question: Is it possible for the end user to select a serverless client image version or is that not something that I can choose? If so, how? I couldn't find any documentation regarding this.

[Louis_Frolio]

Serverless is management free which means you cannot choose the image.  Hope this helps. Lou.

[nija · Accepted solution]

@chinmay0924 - You can change the serverless client image by selecting the environment panel in a Databricks Notebook (on the right pane) or in the "Environment and Libraries" section while configuring a Databricks Job Task.

The set of available serverless client images (officially called Serverless environment versions is here https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/

In the case of Databricks Connect, upgrade to DB Connect 15.4.10 or later (on the 15.x line) or 16.4.1 (on the 16.x line). The reported issue has been fixed in these versions. The compatible Python version on 15.4.10 (or later) is Python 3.11, and on 16.4.1 (or later), it is Python 3.12. 

More details can be found here: https://docs.databricks.com/aws/en/dev-tools/databricks-connect/requirements#databricks-connect-vers...