Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-24-2023 11:14 AM
Thanks a lot for the details. One point I still don't get is the difference between these two points (and let's forget vacuum for this):
- If your streaming query ran 2 weeks ago, it will not reprocess all table versions since then.
- Instead, it will process only the new records introduced during that time period.
Let's say my source delta table version is 2500. I execute a streaming job once with availableNow=True. So it loads everything up to table version 2500.
Now for two weeks I insert, delete and update data in this source table. After 2 weeks, I'm at version 2750. Now I execute the streaming job again.
I don't understand the difference: everything between versions 2500 and 2750 is exactly what has changed? Does the second bullet point mean, it only processes inserts but not deletes and updates?
Thanks for clearifying.