Trouble Enabling File Events For An External Location

[Dnt_TchTheRolex]

Hello all,

I am trying to enable file events on my Azure Workspace for the File Arrival Trigger trigger mode for Databricks Workflows. I'm following this documentation exactly (I think) but I'm not seeing the option to enable them. As you can see here, my Azure Managed Identity has all of the required roles listed in the documentation assigned:

However, when I go to the advanced options of the external location to enable file events, I still do that see that option

In addition, I'm a workspace and account admin and I've granted myself all possible permissions on all of these objects so I doubt that could be the issue. Maybe it's some setting on my storage account or something extra that I have to set up? Any help here/pointing me to the correct documentation would be greatly appreciated

[Stentone]

I'm having this same issue with Databricks for AWS as well. The option does not show up under an existing external location, and I also have admin privileges, so I'm doubting it's a permission issue.

According to the docs, this is a public preview. Perhaps it's one of the few that requires a request for access per docs as I don't see it in our list of previews that can be activated in the workspace.

[BigRoux]

To enable file events for the File Arrival Trigger on your Azure Databricks Workspace, ensure the following requirements and configurations are in place:

1. Azure Managed Identity Roles: Confirm that the Azure Managed Identity used has the necessary roles assigned:

   • Storage Blob Data Contributor
   • EventGrid EventSubscription Contributor
   • Storage Queue Data Contributor (required only if Azure Databricks is expected to create storage queues for you).
2. External Location Permissions: Verify that you are the owner of the external location or have the MANAGE privilege assigned on that external location in Unity Catalog.
3. Unity Catalog Configuration: The workspace must have Unity Catalog enabled, and the external location should be properly configured using Unity Catalog with file events enabled as mentioned in the documentation.
4. Storage Account Configuration:

   • Log into the Azure Data Lake Storage account linked to your external location and validate the IAM permissions for your managed identity using the connector ID or user-assigned managed identity ID obtained from the Catalog > External Locations section in Databricks.
   • Ensure that all required roles have been granted in Azure IAM.
5. Enable File Events in External Location:

   • Navigate to Catalog > External data > External Locations, and select the external location of interest.
   • Click Edit under the kebab menu next to the "Test connection" button.
   • In Advanced Options, enable File events and choose the file event type:
     • Automatic: Allows Databricks to set up subscriptions and events for you.
     • Provided: Requires you to manually configure Azure storage queues and provide the queue URL in the corresponding field.
6. Testing Connection: After enabling file events, wait a few minutes and then use the Test connection functionality to confirm that file events are active on the external location.

If the option to enable file events is still not visible after completing these steps, check for the following potential issues: - Managed identity permissions in Azure IAM might be incorrectly configured, even if they appear complete.

- There may be a limitation or requirement specific to your version of Unity Catalog or Azure Databricks Workspace that is causing the issue (this feature is in Public Preview).

Hope this helps. Cheers, Lou.

[Dnt_TchTheRolex]

1. My managed identity has these roles, as you can see in the screenshot

2. I am the owner and have MANAGE--I'm also the account and workspace admins.

3. Yes, UC is enabled and I followed the documentation exactly as you can see in the screenshots

4. You can see this has been done in the screenshots

5. I don't have this option, as you can see in the last screenshot

6. I don't have the option to enable file events which is what this post is about

[Stentone]

@Dnt_TchTheRolex - This does seem to be a feature that needs to be enabled on the Databricks side after talking with our Databricks solutions engineer. I recommend reaching out to them to see if they can enable it for your account.