Clarification on Auto Loader Managed File Events with Unity Catalog Managed Volumes

[APJESK]

Hi Databricks Team,

I'm trying to understand how Auto Loader Managed File Events work with Unity Catalog Managed Volumes, and I'm looking for some clarification.

My understanding

1. I create an External Location:

CREATE EXTERNAL LOCATION app_ext_loc
URL 's3://my-bucket/catalog/cat1'
WITH (STORAGE CREDENTIAL my_storage_credential);

1. I enable File Events on this External Location.
2. I create a Managed Volume or Managed Table that uses managed storage under this External Location.

The physical storage becomes something like:

s3://my-bucket/catalog/cat1/__unitystorage/catalogs/<catalog-id>/...

My confusion

The documentation says that enabling File Events on an External Location configures S3 Event Notifications for the bucket/prefix and allows Auto Loader to use:

.option("cloudFiles.useManagedFileEvents", "true")

However, Managed Volumes and Managed Tables are stored under the internally managed __unitystorage path.

My questions are:

1. Does Databricks configure S3 Event Notifications for the entire External Location prefix (for example, catalog/cat1/), so events are also generated for objects created under __unitystorage?
2. If yes, can Auto Loader with cloudFiles.useManagedFileEvents=true be used to ingest data from a Managed Volume, or is this feature officially supported only for External Volumes/External Locations?
3. If Managed Volumes are not intended to use Managed File Events, what is the purpose of enabling File Events on an External Location that is also used as the managed storage location for Unity Catalog?
4. Is the recommended architecture:
• External Volume + Managed File Events for ingestion from cloud storage
• Managed Volume + Directory Listing for Databricks-managed storage?

I'm looking for clarification on the intended architecture because the documentation on External Location File Events and Unity Catalog managed storage seems to overlap, and I'd like to understand the supported design pattern.

Thank you.

[Ashwin_DSA]

Hi @APJESK,

I am wondering if the main source of confusion is the name cloudFiles.useManagedFileEvents. In the public docs, "managed" here refers to the Databricks-managed file-events service and cache layer, not to Unity Catalog managed volumes or the __unitystorage path. The Auto Loader with file events overview explains that Databricks sets up cloud resources to listen for file events, processes those notifications, and then Auto Loader reads from that file-events cache when cloudFiles.useManagedFileEvents=true is set.

On your first question... at a high level, file events are configured at the external-location scope. The file notification mode docs say Databricks uses a single file notification queue for all streams from a given external location, and the manage external locations docs say enabling file events on an external location allows Databricks to configure S3-side notifications for that location. So, if your managed storage is physically landing under s3://my-bucket/catalog/cat1/__unitystorage/..., as in your example, then it is still under the broader external-location prefix s3://my-bucket/catalog/cat1/ that you described. Because of that, object creates under __unitystorage would be within that same underlying external-location notification scope. That said, the public docs do not present __unitystorage as the intended Auto Loader ingestion target.

On your second question...I would be careful not to treat that as official support for managed volumes as a managed-file-events source. The public docs consistently describe the source as an external location or a volume on an external location, and they specifically recommend creating a separate external volume for each ingest path and passing the volume path to Auto Loader. I do not see public documentation that explicitly positions managed volumes as an officially supported Auto Loader source for cloudFiles.useManagedFileEvents=true. If you do see that, can you please post the link to that page?

There is also a related signal in the file arrival trigger docs... they explicitly say the monitored path must not contain managed locations of catalogs and schemas. That is a different feature from Auto Loader itself, but it points in the same direction and reinforces that Databricks is careful about treating Unity Catalog managed locations as event-driven file-ingestion sources.

On your third question...the answer is that enabling file events is still valuable for workloads that read paths governed by that external location. The docs say it makes Auto Loader setup easier, enables incremental file discovery with notification-like performance, and improves file-arrival triggers for jobs. In other words, enabling file events on the external location is about improving ingestion and event-driven processing for that external-location-backed storage boundary. It is not the same thing as saying every path underneath that boundary is a recommended Auto Loader source.

Hope this clarifies your questions.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.