Here are some potential steps and considerations to troubleshoot and resolve the issue:

1. Permissions and Configuration:

   • Ensure that the necessary permissions are correctly set up for file notification mode. This includes having the appropriate roles and permissions for the Azure Event Grid and Azure Queue Storage. The required roles include:
     • Contributor: For setting up resources in your storage account.
     • Storage Queue Data Contributor: For performing queue operations.
     • EventGrid EventSubscription Contributor: For performing Event Grid subscription operations.

2. Event Grid Registration:

   • Verify that Event Grid is registered as a Resource Provider in your Azure subscription. If not, you can register it through the Azure portal under the Resource Providers section.

3. File Notification Events:

   • Check if the file notification events are being correctly generated and processed. For ADLS Gen2, Auto Loader listens for the FlushWithClose event for processing a file. Ensure that this event is being triggered correctly.

4. CloudFiles Options:

   • Review the cloudFiles options you have set. Ensure that all necessary options are correctly configured, including cloudFiles.subscriptionId, cloudFiles.tenantId, cloudFiles.clientId, cloudFiles.clientSecret, cloudFiles.resourceGroup, and cloudFiles.queueName.

5. Backfill Interval:

   • Consider setting the cloudFiles.backfillInterval option to trigger regular backfills. This can help ensure that all files are discovered within a given SLA if data completeness is a requirement