Walter_C
Databricks Employee
Databricks Employee

 

  • Yes, moving files from one container to another will reset the file counter for the Databricks File Trigger. The 10,000 file limit applies to the specific storage location being monitored. If files are moved out of this location, they are no longer counted towards the limit, effectively resetting the counter for the new location.

  • If you set up a structure like dir_name/YYYYMMDD for the external location, you will need to change the external location path for each month to ensure the trigger is verified. This is because the file trigger monitors a specific path, and each new month would require a new path to be monitored to stay within the 10,000 file limit.

 

View solution in original post