auto loader
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2024 11:50 AM
i have issue with auto loader that i can't getting incremental load on this . if rerun the ingest() the some unstructured data ingecting into Opportunity_raw
India .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-21-2024 02:51 PM
Hi JissMatew,
As long as your checkpointLocation is not deleted and does not change, you should receive an incremental feed when loading data. Please verify that the checkpoint is not being deleted or moved between runs. The checkpoint is how the stream keeps track of its progress.
If you're still noticing this, please give more details about the duplicate unstructured data that you are ingesting.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-09-2025 11:01 PM
In case of failures, Auto Loader can resume from where it left off by information stored in the checkpoint location and continue to provide exactly-once guarantees when writing data into Delta Lake. You don’t need to maintain or manage any state yourself to achieve fault tolerance or exactly-once semantics.

