Autoloader move file to archive immediately after processing

kumar_soneta — Mon, 02 Jun 2025 08:40:18 GMT

Hi,

We are using autoloader with spark streaming (Databricks: file detection mode) and Want to move files to archive folder from source immediately after processing file. But I cannot reduce retention window beyond 7 days.

Code:

.option("cloudFiles.cleanSource", "move")

.option("cloudFiles.cleanSource.moveDestination", archive_path_monthly)

.option("cloudFiles.cleanSource.retentionDuration", "interval 1 minutes")

Do suggest alternate way to achieve the same

Note: I dont want to do this job via code manualy but I want to configure this with autoloader.

Re: Autoloader move file to archive immediately after processing

vaibhavs120 — Mon, 02 Jun 2025 10:00:04 GMT

cloudFiles.cleanSource.retentionDuration

Type: Interval String

Amount of time to wait before processed files become candidates for archival with cleanSource. Must be greater than 7 days for DELETE. No minimum restriction for MOVE.

Available in Databricks Runtime 16.4 and above. Default value: 30 days

Alternative Solutions :

1. Use Azure storage lifecycle policy.

2. Create a databricks jobs with autocleanup(A delta log tracker is required to make sure the files are processed before moving).

3. Use Azure event grid to trigger a movement operation on file as ingested and an Azure function to listen to the files in the source directory and moves them immediately after autoloader ingestion.

4. Instead of moving files, we can also use databricks external location where the source folder is mapped to a temp directory, and an Azure storage tiering rule automatically moves the files after autoloader ingestion.

topic Re: Autoloader move file to archive immediately after processing in Data Engineering

Autoloader move file to archive immediately after processing

Re: Autoloader move file to archive immediately after processing