Trouble Enabling File Events For An External Location

[v1b3_c0d3rXXX]

Hello all,

I am trying to enable file events on my Azure Workspace for the File Arrival Trigger trigger mode for Databricks Workflows. I'm following this documentation exactly (I think) but I'm not seeing the option to enable them. As you can see here, my Azure Managed Identity has all of the required roles listed in the documentation assigned:

However, when I go to the advanced options of the external location to enable file events, I still do that see that option

In addition, I'm a workspace and account admin and I've granted myself all possible permissions on all of these objects so I doubt that could be the issue. Maybe it's some setting on my storage account or something extra that I have to set up? Any help here/pointing me to the correct documentation would be greatly appreciated

[Stentone]

I'm having this same issue with Databricks for AWS as well. The option does not show up under an existing external location, and I also have admin privileges, so I'm doubting it's a permission issue.

According to the docs, this is a public preview. Perhaps it's one of the few that requires a request for access per docs as I don't see it in our list of previews that can be activated in the workspace.

[Louis_Frolio]

To enable file events for the File Arrival Trigger on your Azure Databricks Workspace, ensure the following requirements and configurations are in place:

1. Azure Managed Identity Roles: Confirm that the Azure Managed Identity used has the necessary roles assigned:

   • Storage Blob Data Contributor
   • EventGrid EventSubscription Contributor
   • Storage Queue Data Contributor (required only if Azure Databricks is expected to create storage queues for you).
2. External Location Permissions: Verify that you are the owner of the external location or have the MANAGE privilege assigned on that external location in Unity Catalog.
3. Unity Catalog Configuration: The workspace must have Unity Catalog enabled, and the external location should be properly configured using Unity Catalog with file events enabled as mentioned in the documentation.
4. Storage Account Configuration:

   • Log into the Azure Data Lake Storage account linked to your external location and validate the IAM permissions for your managed identity using the connector ID or user-assigned managed identity ID obtained from the Catalog > External Locations section in Databricks.
   • Ensure that all required roles have been granted in Azure IAM.
5. Enable File Events in External Location:

   • Navigate to Catalog > External data > External Locations, and select the external location of interest.
   • Click Edit under the kebab menu next to the "Test connection" button.
   • In Advanced Options, enable File events and choose the file event type:
     • Automatic: Allows Databricks to set up subscriptions and events for you.
     • Provided: Requires you to manually configure Azure storage queues and provide the queue URL in the corresponding field.
6. Testing Connection: After enabling file events, wait a few minutes and then use the Test connection functionality to confirm that file events are active on the external location.

If the option to enable file events is still not visible after completing these steps, check for the following potential issues: - Managed identity permissions in Azure IAM might be incorrectly configured, even if they appear complete.

- There may be a limitation or requirement specific to your version of Unity Catalog or Azure Databricks Workspace that is causing the issue (this feature is in Public Preview).

Hope this helps. Cheers, Lou.

[v1b3_c0d3rXXX]

1. My managed identity has these roles, as you can see in the screenshot

2. I am the owner and have MANAGE--I'm also the account and workspace admins.

3. Yes, UC is enabled and I followed the documentation exactly as you can see in the screenshots

4. You can see this has been done in the screenshots

5. I don't have this option, as you can see in the last screenshot

6. I don't have the option to enable file events which is what this post is about

[Stentone]

@v1b3_c0d3rXXX - This does seem to be a feature that needs to be enabled on the Databricks side after talking with our Databricks solutions engineer. I recommend reaching out to them to see if they can enable it for your account.

[Dimitry · Accepted solution]

I managed to resolve it with Microsoft Azure support.

The root cause is that even though everything looks OK by the guide, if something was off while setting up in bricks, the setup may, and likely will, get broken behind the scenes. It will produce cryptic errors for you to chase your own tale.

I faced a similar issue, but with a different error, when re-creating setup in production. The solution below worked there too.

Solution:

1. Enable EventGrid provider in the subscription.

2. Add permissions for the connector to the storage account:

     Note that "Event Grid" one is not appearing in UI but to be assigned by the CLI. UI allows it in the subscription level though. Script below will help you.

3. Go into storage account queues and kill the one created by bricks before:

4. Redo the Databricks connector and external location.

   This is critical. You need to delete and recreate them in bricks.

   Note that you can autofill the resource group from the connector. Be careful, it only works if the connector in the same resource group as the storage account. Otherwise fill the resource group name manually.

5. Test file events in the connector and also create a test job watching the files, to POC the events. 

I have drafted following PS script to assign permission to the storage account:

$connector_id = "<guid>"
$storage_id = "/subscriptions/<sub guid>/resourceGroups/<rg guid>/providers/Microsoft.Storage/storageAccounts/<adls name>"


New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "EventGrid EventSubscription Contributor" -Scope $storage_id

New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "Storage Account Contributor" -Scope $storage_id

New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "Storage Blob Data Contributor" -Scope $storage_id

New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "Storage Queue Data Contributor" -Scope $storage_id

ref:  Databricks - Cannot activate File Events for External Location / ADLS V2 - Microsoft Q&A

[Drupe]

I also have an external location that is a ADLS v2 however from databricks I only want to grant read access to the managed identity. I want to use file arrival trigger but want to have option of using more than 50 such triggers. 
Is it possible to only grant Storage Blob Data Reader and EventGrid EventSubscription Contributor
to enable file events for the external locations? 

[khushboo2002]

I am also facing some issue with the "file arrival" trigger on one of my environments.
Its strange that it kept running fine until today where it broke. 
I always only had the "Storage Blob Data Contributor" for my Managed Identity ( Access Connector) and never had the other ones mentioned above? 
Could you please suggest on why would this work well until now?

I do not see any queues or event subscription being created for my envionments where this passed.

"""
An error occurred while processing file events in the external location associated with '/path/to/Volume': INVALID_STATE: Failed to list objects. There are problems on the location that need to be resolved. Details: Failed to provision file events resources during queue.create operation. Please ensure the IAM roles Storage Account Contributor, Storage Blob Data Contributor, EventGrid EventSubscription Contributor, Storage Queue Data Contributor are correctly assigned to your managed identity (or service principal, if used). Original error from the cloud provider: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call. If you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK.

"""

[Dimitry]

hi, I understand it depends if you use events out of the box, subscribe to your own queue or just look for file changes (this just counts number of files in the path including sub-folders to trigger)

If you listen to the events created by Databricks, you need all ones I mentioned, so it can create queues. I think the role that can do all of it is just "Contibutor", not "Storage Blob Data Contributor", if you go granular you need all mentioned roles.

- If "Enable file events" is disabled, "File Arrival" trigger will just poll file number periodically and won't use queue -> hopefully will work with "Storage Blob Data Contributor"

- if file events enabled, you can test in the external locations / location / test connection. Without these roles or higher admin type role for your connector it won't work.

Which is your case?