Databricks DLT ADLS Access issue

[VigneshJaisanka]

We have a DLT pipeline configure with spn inside the notebook, which was working fine. Now after credentials expiry, we created new one and updated the same in notebook. Now we are pipeline is not able to read from ADLS.

SPN and my UserId is having contributor privilege on the ADLS.

SPN and Access Key config working fine when we use in interactive cluster and we are able to read the data

But in DLT it is throwing below error.

dlt.errors.exceptions.captured.SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501

Below are the steps we tried but still getting the same error.

1. Configured SPN / Access key in notebook, 2. Configured SPN / Access key in DLT pipeline cluster config.

[SP_6721]

Hi @VigneshJaisanka 

The issue likely comes from a permissions or configuration mismatch. Here are a few things worth checking:

• Make sure the SPN is set as the pipeline owner and has the necessary permissions on the ADLS resource.
• If you’re using Unity Catalog, double-check that the SPN has the right privileges (like SELECT) on the relevant catalog, schema, and external tables.
• Confirm that the SPN credentials are correctly set in both the cluster settings and the pipeline settings
• Enabling driver logs might also help pinpoint exactly where the permission is failing.

[VigneshJaisanka]

@SP_6721 
Thanks for the reply.

We setup the SPN as pipeline owner. Still the pipeline is failing.

When we ran the pipeline by setting up admin as the owner of pipeline then the pipeline is running fine. In this case SPN is configured in pipeline configuration.

If SPN is not configured in pipeline obviously even if it admin user it is failing.

Any suggestions from your end would be greatly appreciated.