Autoloader move file to archive immediately after processing
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-02-2025 01:40 AM
Hi,
We are using autoloader with spark streaming (Databricks: file detection mode) and Want to move files to archive folder from source immediately after processing file. But I cannot reduce retention window beyond 7 days.
Code:
.option("cloudFiles.cleanSource", "move")
.option("cloudFiles.cleanSource.moveDestination", archive_path_monthly)
.option("cloudFiles.cleanSource.retentionDuration", "interval 1 minutes")
Do suggest alternate way to achieve the same
Note: I dont want to do this job via code manualy but I want to configure this with autoloader.
