Hi all, I had the same issue (on Windows) and managed to get it working by specifying a user id in the spark remote.

There's the hard-coded way:

from databricks.connect import DatabricksSession 
 
spark = DatabricksSession.builder.remote( "sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>;user_id=123123" ).getOrCreate()

Or with an environment variable:

set SPARK_REMOTE=sc://<databricks_workspace_name>:443;token=<token>;x-databricks-cluster-id=<cluster-id>;user_id=123123

Or similar in python. The actual value (123123) of the user_id can be whatever.