topic Re: Installing Databricks Connect breaks pyspark local cluster mode in Data Engineering

Installing Databricks Connect breaks pyspark local cluster mode

htu — Fri, 03 May 2024 06:14:32 GMT

Hi, It seems that when databricks-connect is installed, pyspark is at the same time modified so that it will not anymore work with local master node. This has been especially useful in testing, when unit tests for spark-related code without any remote session.

Without databricks-connect this code works fine to initialize local spark session:

spark = SparkSession.Builder().master("local[1]").getOrCreate()

However, when databricks-connect python package is installed that same code fails with

> RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

Question: Why does it work like this? Also, is this documented somewhere? I do not see it mentioned in Databricks Connect Troubleshooting or Limitations documentation pages. Same issue has been asked at Running pytest with local spark session · Issue #1152 · databricks/databricks-vscode · GitHub.

Re: Installing Databricks Connect breaks pyspark local cluster mode

htu — Fri, 03 May 2024 09:26:04 GMT

Hi, I undestand Databricks Connect is used for (that why I'm trying it out) but I would also like to be able to run tests. What do you mean with "different local mode"?

As a side-topic, I tried running pytest tests with Databricks Connect session (both spark-connect server running in container at sc://localhost or Azure Databricks via DatabricksSession) and some of the tests fail with "Windows fatal exception: access violation" in both cases so that doesn't really work either.

Re: Installing Databricks Connect breaks pyspark local cluster mode

dmytro — Tue, 27 Aug 2024 19:30:43 GMT

Could you please provide an example of the proposed workaround? Nothing what I have tried helps, there is always the same error as in the original post. This is very frustrating to say the least - the inability to switch properly and natively to a local mode.

I look forward to hearing from you.

Kind regards, Dmytro.

Re: Installing Databricks Connect breaks pyspark local cluster mode

dpires92 — Tue, 10 Sep 2024 19:24:30 GMT

Hey guys. I am facing the same issue. Before databricks connect the unit tests with pytest were working properly.

I´ve tried even to create a newSession() and use pytest-spark library but both approach did not work.

I got the following error "

E RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Use DatabricksSession.builder to create a remote Spark session instead.
E Refer to https://docs.databricks.com/dev-tools/databricks-connect.html on how to configure Databricks Connect.

.venv\Lib\site-packages\pyspark\sql\session.py:552: RuntimeError
===================================================================== short test summary info ======================================================================
ERROR tests/unit/test_functions.py::test_if_function_add_year_month - RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Use DatabricksSession.builder to create a remote Spark session instead.
========================================================================= 1 error in 0.17s ========================================================================= "

Let´s wait a solution. In case I have any news I update the topic here. Cheers.

Re: Installing Databricks Connect breaks pyspark local cluster mode

Angus-Dawson — Mon, 16 Sep 2024 17:10:01 GMT

This is ridiculous. It's absolutely unacceptable as "intended behavior" for a professional software package to clobber the functionality of another package just by being installed.

Re: Installing Databricks Connect breaks pyspark local cluster mode

htu — Tue, 17 Sep 2024 07:45:53 GMT

Indeed. This becomes more obvious when I was looking at the databricks-connect wheel package contents, and it includes also pyspark package. The pyspark inside it is like 9 MB whereas regular pyspark package is over 300 MB. I guess they've only left spark-connect client side parts and removed whole server thing. Makes kind of sense but it should not be done in a way to replace existing package.

I even tried connecting to local (docker-hosted) Spark but it crashes on some test cases.

Re: Installing Databricks Connect breaks pyspark local cluster mode

Kolath — Mon, 28 Oct 2024 16:18:53 GMT

Also frustrated by this behavior. Databricks-connect should not replace the rest of local spark.

Is there any solution to this?

Re: Installing Databricks Connect breaks pyspark local cluster mode

Angus-Dawson — Mon, 28 Oct 2024 18:54:41 GMT

I managed to work around it in Poetry by using optional dependency groups, and then when I want to switch between Databricks Connect and local PySpark functionality I run this:

poetry install --with <group x> --without <group y> --sync

Re: Installing Databricks Connect breaks pyspark local cluster mode

lukany — Tue, 26 Nov 2024 09:07:26 GMT

Hi, we are facing this issue as well, i.e. RuntimeError as reported in this comment. We use the workaround with poetry groups as suggested in this comment.

The workaround introduces unnecessary an non-intuitive complexity to dependency management and provides potential space for introducing errors.

Is there any plan to fix this behaviour?

Re: Installing Databricks Connect breaks pyspark local cluster mode

sean_owen — Mon, 02 Dec 2024 15:35:45 GMT

Databricks-Connect is by design a drop-in replacement for pyspark, essentially. It transparently takes over execution of the Spark parts without change to the program, and is definitely a different 'environment' from local pyspark.

As with any situation where you need to deal with separate software environments, you'd typically have a separate venv for each, and as such could have pyspark in one and databricks-connect in another. Does that not answer this?

Re: Installing Databricks Connect breaks pyspark local cluster mode

noah_sunny — Fri, 13 Dec 2024 08:44:10 GMT

How can I configure the environment in pytest.ini?

I need a local Spark session for unit testing, but I encountered the following error: RuntimeError: Only remote Spark sessions using Databricks Connect are supported.

Re: Installing Databricks Connect breaks pyspark local cluster mode

sean_owen — Fri, 13 Dec 2024 13:34:12 GMT

If you want to run Spark locally, you simply use pyspark!

Re: Installing Databricks Connect breaks pyspark local cluster mode

noah_sunny — Mon, 16 Dec 2024 07:51:19 GMT

I got the error mentioned in the original post due to the installation of Databricks Connect.

Re: Installing Databricks Connect breaks pyspark local cluster mode

mslow — Wed, 18 Dec 2024 17:01:59 GMT

I think in case you're deliberately installing databricks-connect, then you need to handle the local spark session creation.

My issue is that I'm using databricks-dlt package which installs databricks-connect as a dependency. In the latest package version 0.3.0, this breaks pyspark.sql SparkSession creation, forcing me to use DatabricksSparkSession.

To solve this I need to hard code databricks-dlt to previous release version which is not ideal, in case next releases bring new features that I would like to use.

Re: Installing Databricks Connect breaks pyspark local cluster mode

nick — Tue, 14 Jan 2025 03:24:47 GMT

Users should be able to have a single Python environment setup with a single set of Python dependencies specified (in pyproject.toml or similar) and installed, and alternately point their code at either a local or remote Spark cluster simply by changing the URL they pass to `DatabricksSession.builder.remote(...)`.

As far as I can tell, this is not possible. Databricks Connect does not work with a local, open source Spark Connect server (i.e. what you get when you run `sbin/start-connect-server.sh`). And open source Spark Connect does not work with a remote Databricks cluster. And as others have pointed out, installing `databricks-connect` and `pyspark` side-by-side yields a broken Python environment.

No one wants to have a Databricks cluster running 24/7 just so they can run their tests quickly. And having multiple Python environments just to handle these incompatibilities means being forced to abandon modern Python packaging tooling like Poetry and going back to manually wrangling venvs.

Is there a design reason Databricks cannot simply enable Databricks Connect to work with open source Spark Connect servers?

Re: Installing Databricks Connect breaks pyspark local cluster mode

sean_owen — Tue, 14 Jan 2025 03:39:37 GMT

I don't know the details but it's not quite the same 'connect'

But I think you can simply have two venvs if you want - one with Connect and one with local pyspark. You probably do want to treat these as distinct environments; they are different environments. This is what virtual environments are there for IMHO.

Re: Installing Databricks Connect breaks pyspark local cluster mode

nick — Tue, 14 Jan 2025 05:39:38 GMT

I don't see how this works with modern Python packaging tooling and standards.

How would your `pyproject.toml` look to support these multiple environments? How would modern build tools (like Poetry, Pipenv, Hatch, etc.) build/publish your project? How would someone build continuous integration testing?

What I'm understanding is that with databricks-connect you have to basically abandon modern Python packaging if you want to be able to run tests locally. You need to manually maintain multiple `requirements.txt` files and matching venvs, and manually switch between one and the other depending on whether your target is a local Spark cluster or remote one. As you switch back and forth, you'll probably need to futz with your IDE's config so that type and lint checks don't break. And to package your application for deployment, you'll have to build an sdist using a hand-rolled script; no `poetry build` for you.

This all feels very kludgy and outdated. There should be a better way, one that works naturally with modern Python packaging standards.

Re: Installing Databricks Connect breaks pyspark local cluster mode

beliz — Fri, 14 Feb 2025 16:48:33 GMT

I use databricks-connect for local IDE-based (pycharm) development of databrick-jobs. New databricks-connect with DatabricksSession made me a lot of trouble, since I needed to maintain two separate import system for local development, and for job execution on databricks. And this messy solution was suggested by databricks here.

I think i found a workaround to this issue. I haven't tested it deeply, but basic functionality seems to be working.

With this I can create and use SparkSession from my IDE, connecting to remote databricks cluster with DBR15.4 (with Apache Spark 3.5.0).

So my solution is the next:

I've removed databricks-connect package and installed pyspark 3.5.0 instead. To access my remote databricks cluster I use spark-connect and its SPARK_REMOTE env variable, which looks something like this:

SPARK_REMOTE=sc://blahlbah.cloud.databricks.com:443/;token=mytoken;x-databricks-cluster-id=myclusterid

I built the value of the variable is based on this documentation.

After configuring the env var for a python script execution I can use SparkSession as before:

from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate()

Re: Installing Databricks Connect breaks pyspark local cluster mode

solanoam — Sat, 08 Mar 2025 21:48:58 GMT

This issue is very bizarre and was very cumbersome to deal with.. but i think i found a solution i can live with

For me, using 2 venv just complicates to project in a way i an not willing to maintain.
Spark Connect, although promising, lacks in a lot of ares compared to the databricks connect, to name a few:

session needs to be rolled every hour due to the token going bad,
some cluster lifecycles are just not recognized by a regular Spark Connect, like when the cluster is warming up, databricks will wait for the cluster to be ready, while spark connect will raise an exception. One CAN map the different scenarios and build handling logic, but why would he?

As I need this specifically for unit-testing, an I use pytest, I downloaded the original pyspark to a directory on my root folder with:

pip install --target pyspark_unpatched pyspark==X.Y.Z

i created this conftest.py in the root of the project:

def override_databricks_spark() -> None: repo_root = Path(__file__).parent.resolve() unpatched_pyspark_dir = os.path.expanduser(repo_root / "pyspark_unpatched") sys.path.insert(0, unpatched_pyspark_dir) import pyspark as unpatched_pyspark sys.path.remove(unpatched_pyspark_dir) sys.modules["pyspark"] = unpatched_pyspark print(f"Overridden pyspark with module from {unpatched_pyspark_dir}") override_databricks_spark()

Because conftest loads first, It allows you do make sure the the unpatched version of pytest loads first, letting you test peacefully

The main drawback of this is you must use the `pyspark` import statement, using Databricks will break the tests (not that it matters, at least i think it doesn't). and obvoiusly, downloading pyspark, but considering the alternatives, i am willing to bite that bullet.

This was my ref from stack overflow

Re: Installing Databricks Connect breaks pyspark local cluster mode

Martinitus — Fri, 14 Mar 2025 11:41:24 GMT

I agree with most of the comments above, that the current approach of databricks-connect is not great (it sucks to be frankly). Its an issue that was bugging me since more than 2 years now.
By the way, i checked how this could be done with poetry and uv. In poetry i got something somewhat working since this PR was merged: https://github.com/python-poetry/poetry/pull/9553
With uv I still don't have an acceptable solution. In fact the uv devs seem to claim, that there is no way to solve this with the current python packaging specifications: https://github.com/astral-sh/uv/issues/10238#issuecomment-2575893989