Why is my streaming job not resuming even though I specified checkpoint directory

[brickster_2018]

I have provided the checkpointLocation as below, however I see the config is ignored for my streaming query

option("checkpointLocation", "path/to/checkpoint/dir")

[brickster_2018 · Accepted solution]

This is a common question from many users. If the streaming checkpoint directory is specified correctly then this behavior is expected. 

Below is an example of specifying the checkpoint correctly

df.writeStream
 
 .format("parquet")
 
 .option("checkpointLocation", "path/to/checkpoint/dir")
 
 .option("path", "path/to/destination/dir")
 
 .start()

Instead, if you are specifying the checkpointLocation as readStream option it's ignored. One other common scenario where the option is ignored is when you specify the checkpointLocation as an option for the write operation within the forEachBatch. The option should be provided for the writeStream API