Autoloader on ADLS blobs with archival enabled
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2024 08:14 AM
Hi All, I'm trying to change our Ingestion process to use Autoloader to identify new files landing in a directory on ADLS.
The ADLS directory has access tier enabled to archive files older than a certain time period. When I'm trying to set up Autoloader on this directory - it throws an error saying - "Operation failed: "This operation is not permitted on an archived blob."", even though I'm using '.option("cloudFiles.includeExistingFiles", "false")'. I'm trying to understand if I'm overlooking something. Any help is greatly appreciated. Thank you.