What is a Checkpoint in Structured Streaming?

[gowri_databrick]

Hi everyone,

I’m learning about Structured Streaming in Databricks and came across checkpoints.

I understand that checkpoints are used to keep track of the progress of a streaming query, but I’d like to understand their purpose more clearly.

For example, if a streaming pipeline is processing customer transactions and the pipeline stops unexpectedly, how does the checkpoint help the pipeline continue processing from where it stopped?

What is the main purpose of checkpoints, and why are they important in a real-time data pipeline?

Thanks!

[Ashwin_DSA]

Hi @gowri_databrick,

Welcome to the world of Structured Streaming!

Think of a checkpoint as a bookmark for your streaming pipeline. It's a directory on durable storage (such as S3, ADLS, or GCS) where Spark Structured Streaming saves your query's progress after each micro-batch. Specifically, it records:

• Offsets... which records from the source have already been processed.
• Commits... which micro-batches have been successfully written to the sink.
• State... for stateful operations like aggregations or deduplication, the intermediate computation state is saved here too.
• Metadata.... the unique query ID and configuration details.

You enable it by setting the checkpointLocation option on your writeStream:

(df.writeStream
  .option("checkpointLocation", "/Volumes/catalog/schema/volume/checkpoint")
  .toTable("catalog.schema.target_table")
)

Say you have a pipeline reading customer transactions from Kafka and writing them to a Delta Lake table. The pipeline has processed transactions 1 through 10,000 and the checkpoint has recorded that progress. Now the cluster crashes.

When the pipeline restarts, Spark reads the checkpoint and sees: "I already committed everything up to offset 10,000." It picks up right at 10,001... no data is lost, and no transaction gets processed twice. Without a checkpoint, the pipeline would have no memory of what it already did. It would either start from the beginning (duplicating everything) or skip ahead and lose data.

The checkpoint is what gives Structured Streaming its exactly-once processing guarantee. Combined with an idempotent sink like Delta Lake, it ensures every record is processed once and only once, even through failures. This is critical for pipelines where duplicates or missing records have real business consequences, like financial transactions, inventory updates, or customer event tracking.

A few things worth keeping in mind as you build:

• Every streaming query needs its own unique checkpoint location. Never share a checkpoint between two different queries.
• Deleting or changing the checkpoint directory resets the query, so it starts fresh from the beginning.
• Certain changes to your query logic (like modifying stateful operations) are not compatible with an existing checkpoint and require starting with a new one. The Structured Streaming checkpoints docs cover exactly which changes are safe and which are not.

If you are just getting started, the Run your first Structured Streaming workload tutorial walks through a complete example with checkpointing. For production pipelines, also check out Production considerations for Structured Streaming, which covers how to configure automatic restarts so your pipeline recovers from failures without manual intervention.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[balajij8]

@gowri_databrick 

Checkpoints in Structured Streaming serve as the ledger and recovery mechanism for streaming queries. It tracks which data has been processed and successfully written, enabling exactly once processing guarantees. A checkpoint contains several key components: offsets (which records to process), commits (which batches completed successfully) and metadata about the stream. When your customer transactions pipeline processes data, Spark writes the offset before starting a batch and writes the commit after the batch completes. This two-phase approach ensures that if the pipeline crashes mid-batch, Spark knows precisely where to resume.

When an unexpected failure occurs, the checkpoint enables automatic recovery without data loss or duplication. Upon restart, Spark reads the checkpoint and discovers the last committed offset. If a batch started but didn't complete (offset exists but no matching commit), Spark automatically reprocesses that batch. For your transactions example, if the pipeline crashes while processing transactions 1000-1500, the checkpoint shows offset 1000 was started but never committed. On restart, Spark reprocesses from transaction 1000 onward ensuring no transactions are lost or duplicated. This fault tolerance is critical in pipelines where stopping to manually determine restart positions would cause data gaps and operational overhead.

[srini_ve]

@gowri_databrick 

Yes, checkpoints are basically how Structured Streaming remembers where it got to.

For example, imagine a streaming pipeline processing customer transactions:

Source → Structured Streaming → Delta table

Suppose the pipeline has processed transactions up to transaction 10,000, and then the cluster suddenly stops.

The checkpoint stores the progress information, so when the pipeline starts again, it can understand what has already been processed and continue from the appropriate point rather than starting from the beginning.

So, in simple terms:

Checkpoint = the streaming pipeline's saved progress

This is important because in a real-time pipeline, we don't want to lose data or process the same data unnecessarily every time there is a restart or failure.

One important point is that the checkpoint location should be persistent and stable. It shouldn't be stored somewhere that gets deleted when the cluster restarts.

A simple example would be:

10,000 transactions processed → pipeline fails → restart → checkpoint helps resume from the saved progress

That's why checkpoints are an important part of reliable Structured Streaming pipelines.

[Islam_hoti]

Hey @gowri_databrick ,

Think of the checkpoint as a bookmark for your stream.

As your query runs, it writes down two things: which data it has already read from the source, and the state it has built up so far, like running counts or aggregations.

So with your transactions example, say the stream has processed everything up to file 500 and then the cluster dies. When you restart, Spark reads the checkpoint, sees that it got through file 500, and picks up at 501. No gap, and nothing gets processed twice. Without a checkpoint it would either start over from scratch or start from now and silently lose everything in between.

That is really the whole point: exactly once processing across restarts. Streams do not run forever without interruption. Clusters restart, jobs get redeployed, code gets updated. The checkpoint is what makes those interruptions boring instead of a data quality incident.

Two practical things worth knowing. Each streaming query needs its own checkpoint location, because sharing one between two queries will break them. And do not delete the checkpoint directory when a stream misbehaves. It is tempting, but it wipes the bookmark, so you will either reprocess everything or lose data.