Hi Alex,

How are you doing today? As per my understanding, It looks like your Azure Databricks cluster doesn't have the right permissions to access Unity Catalog tables and volumes, even though it works with serverless. This could be because serverless clusters automatically handle identity passthrough, while your SINGLE_USER cluster might not have the necessary permissions. First, check if the user assigned to the cluster has SELECT access on the table and READ access on the volume. Also, verify that the cluster is set up for identity passthrough or has the right Azure storage permissions (like RBAC for ADLS). You can check your table and volume permissions using SHOW GRANTS, and if needed, an admin can grant the required access. If the issue persists, try using a serverless or Unity Catalog-enabled shared cluster to see if it's related to your cluster settings.

Regards,

Brahma