topic Can I reset the checkpoint of a streaming job if I want to do a full reload of a table? in Data Engineering https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21770#M14881 Tue, 22 Jun 2021 15:24:22 GMT User16826992666 2021-06-22T15:24:22Z Can I reset the checkpoint of a streaming job if I want to do a full reload of a table? https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21770#M14881 Tue, 22 Jun 2021 15:24:22 GMT https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21770#M14881 User16826992666 2021-06-22T15:24:22Z Re: Can I reset the checkpoint of a streaming job if I want to do a full reload of a table? https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21771#M14882 <P>Check this out - <A href="https://kb.databricks.com/streaming/ss-read-from-last-offset.html" target="test_blank">https://kb.databricks.com/streaming/ss-read-from-last-offset.html</A></P><P></P><P><A href="https://kb.databricks.com/streaming/file-sink-streaming.html" alt="https://kb.databricks.com/streaming/file-sink-streaming.html" target="_blank">Streaming with File Sink: Problems with recovery if you change checkpoint or output directories</A></P><P></P> Tue, 22 Jun 2021 17:42:01 GMT https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21771#M14882 Digan_Parikh 2021-06-22T17:42:01Z Re: Can I reset the checkpoint of a streaming job if I want to do a full reload of a table? https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21772#M14883 <P>If the read stream definition has something similar to </P><PRE><CODE>val df = spark .read .format("kafka") .option("kafka.bootstrap.servers", "host1:port1,host2:port2") .option("subscribePattern", "topic.*") .option("startingOffsets", "earliest")</CODE></PRE><P>resetting the checkpoint would attempt to read from the earliest record inside the topic. Now, whether this would result in the full reload of the table would be a function of retention.ms of the topic. If there are are records that have already been expired from kafka, they won't be reprocessed. </P><P></P><P></P><P></P> Tue, 22 Jun 2021 22:44:42 GMT https://community.databricks.com/t5/data-engineering/can-i-reset-the-checkpoint-of-a-streaming-job-if-i-want-to-do-a/m-p/21772#M14883 sajith_appukutt 2021-06-22T22:44:42Z