We got our app working by using the databricks-connect and creating a spark session on a shared compute cluster separate from the cluster my app runs on. I set the env variable DATABRICKS_CLUSTER_ID to the id of the cluster I want to use. Then in that cluster (and any catalog you need to access) you can set permission for your apps service principal. You can find the name of your apps service principal in the apps overview screen. You can set the SP to have permission to attach. You must also make sure that the version of databricks your cluster is running and the version of databricks-connect you are using are compatible (I just used version 14.* for both).