topic Re: SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permissio in Data Engineering

SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SE

GANAPATI_HEGDE — Tue, 08 Jul 2025 13:44:15 GMT

I am trying to build a dlt pipeline which fetches data from fabric onelake tables and stores it in databricks.

To do this, i am using azure service principals and databricks asset bundles. i have setup authentication configs as per official documentaion from azure. but i am facing below error even though i have all the necessary acceses

SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501

the code works perfectly fine when i run them in personal compute but throws this error when i run them in job clusters.

Re: SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permissio

nayan_wylde — Tue, 08 Jul 2025 14:13:50 GMT

In the workflow check the Runas. If the workflow is running using an SPN. Check if the SPN have appropriate permissions to read and select files.

Re: SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permissio

GANAPATI_HEGDE — Tue, 08 Jul 2025 15:22:33 GMT

Yes run as is set to SP. SP has permission to onelake, and unity catalog

Re: SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permissio

nayan_wylde — Tue, 08 Jul 2025 17:33:40 GMT

Can you please share the grants that is on the table. I highly doubt there is select permission granted on this schema.