cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Auto Loader authorization exception

v01d
New Contributor III

Hello,
I'm trying to process the DB Auto Loader with notifications=true option (Azure ADLS) and get not clear authorization error. The exception log attached.
Looks like all required permission are provided to the service principle:

Screenshot_2024-06-01_at_14_32_06.png

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @v01d, There can be three probable causes - 

  • The service principal used for authentication lacks the necessary permissions.
    • Confirm that the service principal has the required permissions on the ADLS.
    • Specifically, ensure that it has Read permission on the input folder and Write permission on the output folder.
    • If you’re using a custom role, verify that it includes these permissions.
  • Token expiration or invalid credentials.
    • Check if the access token used by the service principal has expired.
    • If it has, refresh the token programmatically or restart the process that uses the service principal.
  • Network restrictions prevent communication with Azure Data Lake Storage (ADLS).
    • Verify that there are no network restrictions (firewalls, security groups) preventing communication between the service principal and ADLS.
    • If your ADLS is private, ensure that the network configuration allows communication.
    • If you’ve already covered these steps, please share any additional error codes or messages from the logs. We’ll continue troubleshooting until we find a resolution.

v01d
New Contributor III

This service principle has all required permissions, and it successfully reads/writes to this ADLS. But in the cloudFiles.useNotifications=true mode, it throws the exception. I double-checked the required permissions, everything looks good.
I checked also the subscription Resource provider for DataGrid, it's permitted.
The exception log is too common.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!