Can I reset the checkpoint of a streaming job if I want to do a full reload of a table?

[Digan_Parikh]

Check this out - https://kb.databricks.com/streaming/ss-read-from-last-offset.html

Streaming with File Sink: Problems with recovery if you change checkpoint or output directories

[sajith_appukutt · Accepted solution]

If the read stream definition has something similar to

val df = spark
  .read
  .format("kafka")
  .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
  .option("subscribePattern", "topic.*")
  .option("startingOffsets", "earliest")

resetting the checkpoint would attempt to read from the earliest record inside the topic. Now, whether this would result in the full reload of the table would be a function of retention.ms of the topic. If there are are records that have already been expired from kafka, they won't be reprocessed.