Databricks autoloader with manual file delete?

[ctech932]

While we evaluate moving our many autoloader configurations to use `cloudFiles.cleanSource` , we're wondering if we can instead just implement a lifecycle policy outside of Databricks that deletes files older than 30 days.

Is there a problem with doing this? (For example, if the azure storage account lifecycle policy deletes files older than 30 days while autoloader is running, is that a problem?)  (Our autoloader configuration is in directory full listing mode and we have thousands of files a day coming in and only just realized how long autoloader is spending reading over files it has already processed, not to mention the storage costs we're paying).

We're trying to migrate to use the cleanSource option, but in the meantime, it is much faster for us to implement a lifecycle policy across all of our storage accounts. We're wondering if that is a viable solution while we work on migrating to the built-in cleanSource capability.

Thank you~

Nathan

[szymon_dybczak · Accepted solution]

Hi @ctech932 ,

Short answer - yes, you can use an Azure Storage lifecycle policy to delete files older than 30 days.

In directory listing mode, Auto Loader works like this:

1. Lists files in the directory

2. Filters out already-processed files using the checkpoint state

3. Processes new files

4. Updates checkpoint

So the real question is - can you guarantee that all files are processed well within 30 days?

If yes - lifecycle policy is safe.
If no - you risk silent data loss.

For instance if your processing job won't run for 30 days for some reason then your Azure Policy will archive the files that should be loaded. But if you actively monitor your production jobs and you're sure that this won't happen then sure, you can use azure policy instead of auto loader cleanup.

[saurabh18cs]

Hi  why are you not planning to move away from directory listing mode to useManagedFileEvents?? execution will be faster and no more scanning of directories everytime. 

File events use a single Azure Databricks-managed file notification queue for all streams that process files from any given external location defined in Unity Catalog. It is a managed service offered by Databricks on top of external locations in Unity Catalog. It provides the following benefits

• Makes it easier to set up file notifications for Auto Loader. Specifically, it enables incremental file discovery with notification-like performance in Auto Loader. 
• Improves the efficiency and capacity of file arrival triggers for jobs.

This solves

• Pain points around long startup times for DLT and Auto Loader
• Enable Auto Loader to read from shared Unity Catalog locations
• Scale file arrival triggers in jobs to storage of arbitrary size
• Run many streams reading from the same storage
• No longer need to create a queue for each Auto Loader stream, it's easier to avoid hitting the cloud provider notification limit

When first enabling the file even notification on an external location, Databricks will do a full scan on the external location to discover all files that exists and store it in the managed file events service. Notifications that are consumed and stored by the managed file events service or automatically removed from the cloud queue. The notifications will be stored in the managed service for 7 days. If autoloader does not consume the notifications within that period, it will fallback to directory listing to guarantee completeness.