Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-21-2024 07:27 AM
Hello @Devsql
Autoloader initially lists files using one of the File Detection Modes. For each batch of files discovered, a checkpoint is created. If you wish to examine the state of your checkpoint, you can use the cloud_files_state SQL function, which displays all files discovered by Autoloader.
Autoloader uses Checkpointing to maintain state, maintaining exactly-once processing guarantees throughout your spark structured streaming query.
I hope my answer is helpful to you, I've attempted to provide a comprehensive overview of Databricks Autoloader's features.
Best regards,
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks