Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-16-2025 12:37 AM
Hi @HW413 ,
You won't find checkpoint. COPY INTO does not use checkpoint like autoloader or spark structured streaming.
The COPY INTO command retrieves metadata about all files in the specified source directory/prefix . So, every time you run copy into, the command first will create in-memory index of all files. You can see it yourself in SPARK UI:
Then it will compare which files has been already loaded using delta log and this in-memory index of files build in previous step.