Hi @mjtd .

The com.databricks.sql.util.UnexpectedHttpStatus: Failed to list objects error during migration to File Events mode typically indicates one of two things: either the External Location hasn't had File Events enabled yet in Unity Catalog, or the Databricks Access Connector is missing one or more required Azure RBAC roles.

Step 1 — Enable File Events on the External Location first

Before setting cloudFiles.useManagedFileEvents=True in your stream, you must explicitly enable File Events on the External Location in Unity Catalog. Without this, the useManagedFileEvents flag will fail at the infrastructure setup phase with SQL or via the Unity Catalog UI: Catalog Explorer → External Locations → (your location) → Edit → Enable File Events. You can verify the setup is correct by clicking Test Connection — look for a green checkmark on the File Events Read item.

Step 2 — Assign the correct RBAC roles to the Access Connector

The Access Connector for Azure Databricks needs the following Azure roles (in addition to Storage Blob Data Contributor which you may already have):

Step 3 — Register the EventGrid resource provider

If this is your first time using File Events in the subscription, make sure Microsoft.EventGrid is registered.

Step 4 — Stop the old stream, tear down legacy notification resources, then restart

Per the official migration guide, you should:

1. Stop the existing Auto Loader stream
2. Delete the legacy Azure Event Grid subscription and Storage Queue created by the old cloudFiles.useNotifications=True mode
3. Keep the same checkpoint directory — it will be reused
4. Restart the stream with cloudFiles.useManagedFileEvents=True (no need for cloudFiles.useNotifications=True)

df = (spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "parquet") # or json/csv/etc.
.option("cloudFiles.useManagedFileEvents", "true") # the new flag
.load("<path>")
)

Let me know if this can help you and If this helped you, please give it a 👍 Kudo — it helps others find the answer too!