- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-08-2022 11:56 AM
I just tested it out and my stream initialization times seem to have gone down. Can someone explain the backfill interval?
Based on the documentation located here its sounds like the backfill is almost like a full directory list at a given interval to make sure that all files have been processed.
Is this true?
If it is true then that would imply that the option is not used unless the auto loader is configured for event notification services right?
EDIT:
After I let it run for a little while after re starting the stream recs/sec tanked. I went from ~180 rec/esc to ~30 rec/sec.
I have auto compaction and auto optimization turned on as well because I dont have a partitioning column that lets me create balanced partitions.