Dimitry
Valued Contributor

I managed to resolve it with Microsoft Azure support.

The root cause is that even though everything looks OK by the guide, if something was off while setting up in bricks, the setup may, and likely will, get broken behind the scenes. It will produce cryptic errors for you to chase your own tale.

I faced a similar issue, but with a different error, when re-creating setup in production. The solution below worked there too.

Solution:

1. Enable EventGrid provider in the subscription.

2. Add permissions for the connector to the storage account:

Dimitry_0-1754351605646.png

     Note that "Event Grid" one is not appearing in UI but to be assigned by the CLI. UI allows it in the subscription level though. Script below will help you.

3. Go into storage account queues and kill the one created by bricks before:

Dimitry_1-1754351844946.png

4. Redo the Databricks connector and external location.

   This is critical. You need to delete and recreate them in bricks.

   Note that you can autofill the resource group from the connector. Be careful, it only works if the connector in the same resource group as the storage account. Otherwise fill the resource group name manually.

5. Test file events in the connector and also create a test job watching the files, to POC the events. 

I have drafted following PS script to assign permission to the storage account:

$connector_id = "<guid>"
$storage_id = "/subscriptions/<sub guid>/resourceGroups/<rg guid>/providers/Microsoft.Storage/storageAccounts/<adls name>"


New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "EventGrid EventSubscription Contributor" -Scope $storage_id

New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "Storage Account Contributor" -Scope $storage_id

New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "Storage Blob Data Contributor" -Scope $storage_id

New-AzRoleAssignment -ObjectId $connector_id -RoleDefinitionName "Storage Queue Data Contributor" -Scope $storage_id

ref:  Databricks - Cannot activate File Events for External Location / ADLS V2 - Microsoft Q&A

View solution in original post