[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file

[rachelh]

Just wondering if anyone could help me understand why we are hitting this error: `[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file`

A job is trying to create a table with an external location (already setup) using serverless compute. The job was able to do this fine on a standard cluster but once we switched to serverless, we encountered this error. The job is also being "run as" a service principal that has "CREATE EXTERNAL TABLE",  "READ FILES" and "WRITE FILES" on the external location. It appears as though the serverless compute appears to be missing these permissions and falling back to table access control, hence looking for "any file" permissions; can someone help me confirm this? I would like to know why the Unity Catalog permissions are not being honored in this case.

[Khaja_Zaffer]

Hello @rachelh 

Good day!!

serverless compute enforces stricter rules: serverless can't bypass UC for external storage and defaults to an insufficient privilege check under the hood.

The key missing permission is the CREATE TABLE privilege on the schema where you are trying to create the table, along with the correct permissions on the external location. More specifically, the service principal needs to have: CREATE TABLE privilege on the target schema. USAGE privilege on the catalog and schema.

[rachelh]

Hmmm, I do have those permissions assigned to the schema already and appropriate permissions on the external location as stated in the original post.

[BS_THE_ANALYST]

@rachelh this is a nice article around external locations and Unity Catalog: https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/#how-does-unity-catalog-gover... & https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/#external-locations 🙂

All the best,
BS

[saurabh18cs]

Hi @rachelh 

As I understand , you need to look for azure access connector setup for your unity catalog because Serverless clusters run under a Azure Databricks-managed identity, not the service principal.

• Access Connector (Azure Managed Identity): Used by Databricks serverless compute to access storage on behalf of Unity Catalog.

[rachelh]

Sorry, I forgot to mention that we are using AWS as our cloud provider and grant access to S3 using external locations and storage credentials. Your solution may not be relevant to us?