Hey @databricks1111,

thanks for the extra details 

The behavior you’re seeing (works fine on personal compute but fails on shared compute) usually comes down to which identity Databricks uses to access Azure Storage.

When you use personal compute, operations run under your user identity in Unity Catalog. But with shared compute, Databricks uses the workspace’s managed identity (via the access connector).

That difference can cause exactly what you’re seeing the shared compute can list blobs (since it can talk to the storage account) but fails to read the actual files, because the
managed identity tied to the shared compute doesn’t have the same Unity Catalog or storage-level permissions to access the data path.

Here’s what I’d suggest checking:

1. In Azure, make sure the access connector’s managed identity has Storage Blob Data Contributor on both the storage account and the specific container.
2. In Databricks, confirm that the external location’s credential uses this same managed identity.
3. Also verify that the shared compute cluster is Unity Catalog-enabled and has the right cluster access mode (Single User vs Shared Access).

If it’s in Shared mode, ensure users or service principals accessing the files have permission in Unity Catalog to read that external location.