Clarification on Auto Loader Managed File Events with Unity Catalog Managed Volumes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hi Databricks Team,
I'm trying to understand how Auto Loader Managed File Events work with Unity Catalog Managed Volumes, and I'm looking for some clarification.
My understanding
- I create an External Location:
CREATE EXTERNAL LOCATION app_ext_loc
URL 's3://my-bucket/catalog/cat1'
WITH (STORAGE CREDENTIAL my_storage_credential);
- I enable File Events on this External Location.
- I create a Managed Volume or Managed Table that uses managed storage under this External Location.
The physical storage becomes something like:
s3://my-bucket/catalog/cat1/__unitystorage/catalogs/<catalog-id>/...
My confusion
The documentation says that enabling File Events on an External Location configures S3 Event Notifications for the bucket/prefix and allows Auto Loader to use:
.option("cloudFiles.useManagedFileEvents", "true")
However, Managed Volumes and Managed Tables are stored under the internally managed __unitystorage path.
My questions are:
- Does Databricks configure S3 Event Notifications for the entire External Location prefix (for example, catalog/cat1/), so events are also generated for objects created under __unitystorage?
- If yes, can Auto Loader with cloudFiles.useManagedFileEvents=true be used to ingest data from a Managed Volume, or is this feature officially supported only for External Volumes/External Locations?
- If Managed Volumes are not intended to use Managed File Events, what is the purpose of enabling File Events on an External Location that is also used as the managed storage location for Unity Catalog?
- Is the recommended architecture:
- External Volume + Managed File Events for ingestion from cloud storage
- Managed Volume + Directory Listing for Databricks-managed storage?
I'm looking for clarification on the intended architecture because the documentation on External Location File Events and Unity Catalog managed storage seems to overlap, and I'd like to understand the supported design pattern.
Thank you.