I think in case you're deliberately installing databricks-connect, then you need to handle the local spark session creation.

My issue is that I'm using databricks-dlt package which installs databricks-connect as a dependency. In the latest package version 0.3.0, this breaks pyspark.sql SparkSession creation, forcing me to use DatabricksSparkSession.

To solve this I need to hard code databricks-dlt to previous release version which is not ideal, in case next releases bring new features that I would like to use.