Unit Testing with the new Databricks Connect in Python

[cosminsanda]

I would like to create a regular PySpark session in an isolated environment against which I can run my Spark based tests. I don't see how that's possible with the new Databricks Connect. I'm going in circles here, is it even possible?

I don't want to connect to some cluster or anywhere really. I want to be able to run my tests as per usual, without access to internet.

[-werners-]

I'd not use databricks connect/spark connect in that case.
Instead run spark locally.  Of course you will not have databricks specific tools (like dbutils etc)

[cosminsanda]

Problem is that I don't see how you can have both spark native and Databricks Connect (Spark Connect). The guidelines suggest one or the other, which is a bit of a pickle.

[-werners-]

you could try to separate the environments f.e. using containers/vm's. 
Probably there are other ways too, but these immediately came to mind.

[cosminsanda · Accepted solution]

Ok, so the best solution as it stands today (for me personally at least) is this:

• Have pyspark ^3.4 installed with the connect extra feature.
• My unit tests then don't have to change at all, as they use the regular spark session created on the fly
• For running the script locally while taking advantage of Databricks, I use the open source Spark Connect and then set the SPARK_REMOTE=sc://${WORKSPACE_INSTANCE_NAME}:443/;token=${PERSONAL_ACCESS_TOKEN};x-databricks-cluster-id=${CLUSTER_ID}