VZLA
Databricks Employee
Databricks Employee

Thanks for your question! 

If the access connector is still prompting for a secret or certificate when used in cloudFiles.clientId, this typically indicates that the authentication method is not being properly recognized. Here's what to check:

  • Access Connector Role: Ensure the access connector has the correct role assigned in Azure, such as "Storage Queue Data Contributor" for file notifications. Missing roles could cause authentication fallback to expect a client secret.
  • Databricks Runtime Compatibility: Verify that your Databricks runtime version supports using the access connector for file notifications in Auto Loader. Older runtime versions may not fully support this feature.
  • Auto Loader Options: Double-check that no conflicting options, such as cloudFiles.clientSecret or cloudFiles.clientCertificate, are inadvertently set in the configuration, as this could override the access connector's authentication.
  • Connector Permissions: Confirm that the access connector has explicit permissions on the storage queue and the container where file events are stored.

If these checks are all in place and it still fails, it may require additional debugging with Databricks support to ensure compatibility with the current setup.