Databricks File Trigger Limit

[shubhamM]

For Databricks File Trigger below limitation is mentioned.

• A storage location configured for a file arrival trigger can contain only up to 10,000 files. Locations with more files cannot be monitored for new file arrivals. If the configured storage location is a subpath of a Unity Catalog external location or volume, the 10,000 file limit applies to the subpath and not the root of the storage location. For example, the root of the storage location can contain more than 10,000 files across its subdirectories, but the configured subdirectory must not exceed the 10,000 file limit.

1. Does this mean if the files are moved from one container to another it will reset the file counter?

2. If we have to setup structure like dir_name/YYYYMMDD structure for external location. Do we have to change external location path for each month for triggered to be verified.

[Walter_C · Accepted solution]

• Yes, moving files from one container to another will reset the file counter for the Databricks File Trigger. The 10,000 file limit applies to the specific storage location being monitored. If files are moved out of this location, they are no longer counted towards the limit, effectively resetting the counter for the new location.

• If you set up a structure like dir_name/YYYYMMDD for the external location, you will need to change the external location path for each month to ensure the trigger is verified. This is because the file trigger monitors a specific path, and each new month would require a new path to be monitored to stay within the 10,000 file limit.