Just wondering if anyone could help me understand why we are hitting this error: `[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission MODIFY on any file`
A job is trying to create a table with an external location (already setup) using serverless compute. The job was able to do this fine on a standard cluster but once we switched to serverless, we encountered this error. The job is also being "run as" a service principal that has "CREATE EXTERNAL TABLE", "READ FILES" and "WRITE FILES" on the external location. It appears as though the serverless compute appears to be missing these permissions and falling back to table access control, hence looking for "any file" permissions; can someone help me confirm this? I would like to know why the Unity Catalog permissions are not being honored in this case.