Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-24-2021 06:54 AM
I have ad-hoc one-time streaming queries where I believe checkpoint won't give any value add. Should I still use checkpointing
Labels:
- Labels:
-
Structured streaming
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-24-2021 06:57 AM
It's not mandatory. But the strong recommendation is to use Checkpointing for Streaming irrespective of your use case. This is because the default checkpoint location can get a lot of files over time as there is no graceful guaranteed cleaning in place if you do not specify a streaming checkpoint location. Having these stale checkpoint files can impact the performance of queries as file listing on these directories turn out to be very expensive.