topic Re: Unit Testing with the new Databricks Connect in Python in Data Engineering

Unit Testing with the new Databricks Connect in Python

cosminsanda — Tue, 16 Apr 2024 20:48:23 GMT

I would like to create a regular PySpark session in an isolated environment against which I can run my Spark based tests. I don't see how that's possible with the new Databricks Connect. I'm going in circles here, is it even possible?

I don't want to connect to some cluster or anywhere really. I want to be able to run my tests as per usual, without access to internet.

Re: Unit Testing with the new Databricks Connect in Python

-werners- — Wed, 17 Apr 2024 07:28:40 GMT

I'd not use databricks connect/spark connect in that case.
Instead run spark locally. Of course you will not have databricks specific tools (like dbutils etc)

Re: Unit Testing with the new Databricks Connect in Python

cosminsanda — Wed, 17 Apr 2024 08:38:28 GMT

Problem is that I don't see how you can have both spark native and Databricks Connect (Spark Connect). The guidelines suggest one or the other, which is a bit of a pickle.

Re: Unit Testing with the new Databricks Connect in Python

-werners- — Wed, 17 Apr 2024 10:18:48 GMT

you could try to separate the environments f.e. using containers/vm's.
Probably there are other ways too, but these immediately came to mind.

Re: Unit Testing with the new Databricks Connect in Python

cosminsanda — Thu, 18 Apr 2024 08:16:42 GMT

Ok, so the best solution as it stands today (for me personally at least) is this:

Have pyspark ^3.4 installed with the connect extra feature.
My unit tests then don't have to change at all, as they use the regular spark session created on the fly
For running the script locally while taking advantage of Databricks, I use the open source Spark Connect and then set the SPARK_REMOTE=sc://${WORKSPACE_INSTANCE_NAME}:443/;token=${PERSONAL_ACCESS_TOKEN};x-databricks-cluster-id=${CLUSTER_ID}

Re: Unit Testing with the new Databricks Connect in Python

thibault — Tue, 28 May 2024 06:51:51 GMT

Given this doesn't work on serverless compute, aren't those tests very slow to complete due to the compute startup time? I'm trying to steer away from databricks connect for unit testing for this reason. If they supported serverless, that would be a different story.