What is a Checkpoint in Structured Streaming?

[gowri_databrick]

Hi everyone,

I’m learning about Structured Streaming in Databricks and came across checkpoints.

I understand that checkpoints are used to keep track of the progress of a streaming query, but I’d like to understand their purpose more clearly.

For example, if a streaming pipeline is processing customer transactions and the pipeline stops unexpectedly, how does the checkpoint help the pipeline continue processing from where it stopped?

What is the main purpose of checkpoints, and why are they important in a real-time data pipeline?

Thanks!

[Ashwin_DSA]

Hi @gowri_databrick,

Welcome to the world of Structured Streaming!

Think of a checkpoint as a bookmark for your streaming pipeline. It's a directory on durable storage (such as S3, ADLS, or GCS) where Spark Structured Streaming saves your query's progress after each micro-batch. Specifically, it records:

• Offsets... which records from the source have already been processed.
• Commits... which micro-batches have been successfully written to the sink.
• State... for stateful operations like aggregations or deduplication, the intermediate computation state is saved here too.
• Metadata.... the unique query ID and configuration details.

You enable it by setting the checkpointLocation option on your writeStream:

(df.writeStream
  .option("checkpointLocation", "/Volumes/catalog/schema/volume/checkpoint")
  .toTable("catalog.schema.target_table")
)

Say you have a pipeline reading customer transactions from Kafka and writing them to a Delta Lake table. The pipeline has processed transactions 1 through 10,000 and the checkpoint has recorded that progress. Now the cluster crashes.

When the pipeline restarts, Spark reads the checkpoint and sees: "I already committed everything up to offset 10,000." It picks up right at 10,001... no data is lost, and no transaction gets processed twice. Without a checkpoint, the pipeline would have no memory of what it already did. It would either start from the beginning (duplicating everything) or skip ahead and lose data.

The checkpoint is what gives Structured Streaming its exactly-once processing guarantee. Combined with an idempotent sink like Delta Lake, it ensures every record is processed once and only once, even through failures. This is critical for pipelines where duplicates or missing records have real business consequences, like financial transactions, inventory updates, or customer event tracking.

A few things worth keeping in mind as you build:

• Every streaming query needs its own unique checkpoint location. Never share a checkpoint between two different queries.
• Deleting or changing the checkpoint directory resets the query, so it starts fresh from the beginning.
• Certain changes to your query logic (like modifying stateful operations) are not compatible with an existing checkpoint and require starting with a new one. The Structured Streaming checkpoints docs cover exactly which changes are safe and which are not.

If you are just getting started, the Run your first Structured Streaming workload tutorial walks through a complete example with checkpointing. For production pipelines, also check out Production considerations for Structured Streaming, which covers how to configure automatic restarts so your pipeline recovers from failures without manual intervention.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[balajij8]

@gowri_databrick 

Checkpoints in Structured Streaming serve as the ledger and recovery mechanism for streaming queries. It tracks which data has been processed and successfully written, enabling exactly once processing guarantees. A checkpoint contains several key components: offsets (which records to process), commits (which batches completed successfully) and metadata about the stream. When your customer transactions pipeline processes data, Spark writes the offset before starting a batch and writes the commit after the batch completes. This two-phase approach ensures that if the pipeline crashes mid-batch, Spark knows precisely where to resume.

When an unexpected failure occurs, the checkpoint enables automatic recovery without data loss or duplication. Upon restart, Spark reads the checkpoint and discovers the last committed offset. If a batch started but didn't complete (offset exists but no matching commit), Spark automatically reprocesses that batch. For your transactions example, if the pipeline crashes while processing transactions 1000-1500, the checkpoint shows offset 1000 was started but never committed. On restart, Spark reprocesses from transaction 1000 onward ensuring no transactions are lost or duplicated. This fault tolerance is critical in pipelines where stopping to manually determine restart positions would cause data gaps and operational overhead.