Can't access abfss data in azure databricks when providing shared key (fighting UC?)

[DB1To3]

The "no isolation shared" clusters are going to be killed in the next month.  This came as a surprise to me.  I should have been paying closer attention.  We don't heavily use UC since our data is published to the business via Fabric.  All the spark workloads in databricks are totally independent of our data governance (they run as trusted service principals).

The first change I made is to run the spark clusters as the trusted service principal.  (SINGLE USER).  That fails.  I get an error like so, when reading data from abfss: locations:

[Error] [JvmBridge] com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException: PERMISSION_DENIED: User does not have READ FILES on External Location 'xyz-datalake'.

IMO, this is a silly error to receive, if I'm using spark.conf.set() to provide the related shared storage key (spark.hadoop.fs.azure.account.key.whatever).

I think the reason this is happening is because someone else, in the same UC metastore, has configured this storage container to be a UC "external location".  Be that as it may, it shouldn't cause errors if I'm giving the spark session all the necessary credentials by way of spark.hadoop.fs.azure.account.key.

NOTE: The same spark code can be executed from an OSS spark cluster running on-prem without any problems.  Why am I getting errors when running the job in the databricks SaaS?  There is some sort of incompatibility, and I haven't found the config that makes spark run normally in databricks.

[DB1To3]

Here is the full stack trace.  We can see that the reading of parquet from a dataframe almost immediately places dependencies on proprietary databricks code.  It starts relying on UC managed catalog for authn/authz. 

[DB1To3 · Accepted solution]

I found another reply where the customer was required to add the service principal's full GRANT to the external locations.   Seems very odd, and almost less secure than what I was doing with the shared access key.

I really think there needs to be a new session configuration in spark to disable/bypass the bells-and-whistles of the UC governance.  I believe there may already be a "fallback mode", but it doesn't look like I can enable that mode from within my spark workloads themselves.  

At the end of the day, this is only a minor issue when it comes to UC.   There are bigger concerns.  When is there going to be a UC v.2.0?  I think this has been around for a few years now, and I'd really love it if Databricks circled back and fixed some of the many limitations.