https://community.databricks.com/t5/data-engineering/why-is-my-streaming-job-not-resuming-even-though-i-specified/m-p/19843#M13372 <P>This is a common question from many users. If the streaming checkpoint directory is specified correctly then this behavior is expected.&nbsp;</P><P></P><P>Below is an example of specifying the checkpoint correctly</P><P></P><PRE><CODE>df.writeStream &nbsp; .format("parquet") &nbsp; .option("checkpointLocation", "path/to/checkpoint/dir") &nbsp; .option("path", "path/to/destination/dir") &nbsp; .start()</CODE></PRE><P></P><P></P><P></P><P>Instead, if you are specifying the checkpointLocation as readStream option it's ignored. One other common scenario where the option is ignored is when you specify the checkpointLocation as an option for the write operation within the forEachBatch.&nbsp;The option should be provided for the writeStream API</P> Fri, 25 Jun 2021 13:34:46 GMT brickster_2018 2021-06-25T13:34:46Z https://community.databricks.com/t5/data-engineering/why-is-my-streaming-job-not-resuming-even-though-i-specified/m-p/19842#M13371 <P>I have provided the checkpointLocation as below, however I see the config is ignored for my streaming query</P><P></P><P><B><I>option("checkpointLocation", "path/to/checkpoint/dir")</I></B></P><P></P> Fri, 25 Jun 2021 13:33:55 GMT https://community.databricks.com/t5/data-engineering/why-is-my-streaming-job-not-resuming-even-though-i-specified/m-p/19842#M13371 brickster_2018 2021-06-25T13:33:55Z https://community.databricks.com/t5/data-engineering/why-is-my-streaming-job-not-resuming-even-though-i-specified/m-p/19843#M13372 <P>This is a common question from many users. If the streaming checkpoint directory is specified correctly then this behavior is expected.&nbsp;</P><P></P><P>Below is an example of specifying the checkpoint correctly</P><P></P><PRE><CODE>df.writeStream &nbsp; .format("parquet") &nbsp; .option("checkpointLocation", "path/to/checkpoint/dir") &nbsp; .option("path", "path/to/destination/dir") &nbsp; .start()</CODE></PRE><P></P><P></P><P></P><P>Instead, if you are specifying the checkpointLocation as readStream option it's ignored. One other common scenario where the option is ignored is when you specify the checkpointLocation as an option for the write operation within the forEachBatch.&nbsp;The option should be provided for the writeStream API</P> Fri, 25 Jun 2021 13:34:46 GMT https://community.databricks.com/t5/data-engineering/why-is-my-streaming-job-not-resuming-even-though-i-specified/m-p/19843#M13372 brickster_2018 2021-06-25T13:34:46Z