https://community.databricks.com/t5/data-engineering/is-it-mandatory-to-checkpoint-my-streaming-query/m-p/20761#M14055 <P>I have ad-hoc one-time&nbsp;streaming queries where I believe checkpoint won't give any value add. Should I still use checkpointing </P> Thu, 24 Jun 2021 13:54:39 GMT brickster_2018 2021-06-24T13:54:39Z https://community.databricks.com/t5/data-engineering/is-it-mandatory-to-checkpoint-my-streaming-query/m-p/20761#M14055 <P>I have ad-hoc one-time&nbsp;streaming queries where I believe checkpoint won't give any value add. Should I still use checkpointing </P> Thu, 24 Jun 2021 13:54:39 GMT https://community.databricks.com/t5/data-engineering/is-it-mandatory-to-checkpoint-my-streaming-query/m-p/20761#M14055 brickster_2018 2021-06-24T13:54:39Z https://community.databricks.com/t5/data-engineering/is-it-mandatory-to-checkpoint-my-streaming-query/m-p/20762#M14056 <P>It's not mandatory. But the strong recommendation is to use Checkpointing for Streaming irrespective of your use case. This is because the default checkpoint location can get a lot of files over time as there is no graceful guaranteed cleaning in place if you do not specify a streaming checkpoint location. Having these stale checkpoint files can impact the performance of queries as file listing on these directories turn out to be very expensive. </P> Thu, 24 Jun 2021 13:57:33 GMT https://community.databricks.com/t5/data-engineering/is-it-mandatory-to-checkpoint-my-streaming-query/m-p/20762#M14056 brickster_2018 2021-06-24T13:57:33Z