Is it mandatory to checkpoint my streaming query.

brickster_2018
Databricks Employee
Databricks Employee

I have ad-hoc one-time streaming queries where I believe checkpoint won't give any value add. Should I still use checkpointing

brickster_2018
Databricks Employee
Databricks Employee

It's not mandatory. But the strong recommendation is to use Checkpointing for Streaming irrespective of your use case. This is because the default checkpoint location can get a lot of files over time as there is no graceful guaranteed cleaning in place if you do not specify a streaming checkpoint location. Having these stale checkpoint files can impact the performance of queries as file listing on these directories turn out to be very expensive.

View solution in original post