Running `pyspark` with `databricks-connect`

agagrins
New Contributor III

Hiya,

I'm trying to run `pyspark` with `databricks-connect==11.30.b0`, but am failing.

The trace I see is

```

 File "/home/agagrins/databricks9/lib/python3.9/site-packages/py4j/java_gateway.py", line 1321, in __call__

  return_value = get_return_value(

 File "/home/agagrins/databricks9/lib/python3.9/site-packages/pyspark/sql/utils.py", line 196, in deco

  return f(*a, **kw)

 File "/home/agagrins/databricks9/lib/python3.9/site-packages/py4j/protocol.py", line 326, in get_return_value

  raise Py4JJavaError(

py4j.protocol.Py4JJavaError: An error occurred while calling o33.sql.

: org.apache.spark.SparkException: There is no Credential Scope.

    at com.databricks.unity.UCSDriver$Manager.$anonfun$currentScopeId$1(UCSDriver.scala:94)

    at scala.Option.getOrElse(Option.scala:189)

    at com.databricks.unity.UCSDriver$Manager.currentScopeId(UCSDriver.scala:94)

    at com.databricks.unity.UCSDriver$Manager.currentScope(UCSDriver.scala:97)

    at com.databricks.unity.UnityCredentialScope$.currentScope(UnityCredentialScope.scala:100)

    at com.databricks.unity.UnityCredentialScope$.getCredentialManager(UnityCredentialScope.scala:128)

    at com.databricks.unity.CredentialManager$.getUnityApiTokenOpt(CredentialManager.scala:456)

    at com.databricks.unity.UnityCatalogClientHelper$.getToken(UnityCatalogClientHelper.scala:34)

    at com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$getCatalog$1(ManagedCatalogClientImpl.scala:163)

    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)

    at com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapException$1(ManagedCatalogClientImpl.scala:2904)

    at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException(ErrorDetailsHandler.scala:25)

    at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException$(ErrorDetailsHandler.scala:23)

    at com.databricks.managedcatalog.ManagedCatalogClientImpl.wrapServiceException(ManagedCatalogClientImpl.scala:77)

    at com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapException(ManagedCatalogClientImpl.scala:2903)

    at com.databricks.managedcatalog.ManagedCatalogClientImpl.getCatalog(ManagedCatalogClientImpl.scala:156)

    at com.databricks.sql.managedcatalog.ManagedCatalogCommon.catalogExists(ManagedCatalogCommon.scala:94)

    at com.databricks.sql.managedcatalog.PermissionEnforcingManagedCatalog.catalogExists(PermissionEnforcingManagedCatalog.scala:177)

    at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.catalogExists(ManagedCatalogSessionCatalog.scala:384)

    at com.databricks.sql.DatabricksCatalogManager.isCatalogRegistered(DatabricksCatalogManager.scala:104)

    at org.apache.spark.sql.SparkServiceCatalogV2Handler$.catalogOperationV2(SparkServiceCatalogV2Handler.scala:58)

    at com.databricks.service.SparkServiceImpl$.$anonfun$catalogOperationV2$1(SparkServiceImpl.scala:165)

```

I've tried to Google "There is no Credential Scope", but to no avail. Anyone have a clue of where to start to look?