A bad deployment slips into your pipeline on Friday, and by Monday, it has been writing incorrect data for three days, with no clean way to roll it back. You can restore one table, but a pipeline is a graph of tables, source positions, and operator state that all moved forward together, and nothing puts them back in agreement. So most teams reprocess from the beginning, burning compute to fix a three-day mistake.
Today, we're announcing the beta of SDP Rewind, an undo button for Apache Spark™ Declarative Pipelines (SDP) that rolls the whole pipeline back to a consistent point in a single operation. You then fix your code and restart, and the pipeline picks up from the rewind point, so you pay only for the window you got wrong.
Why is this hard today?
Take that Friday deploy. The pipeline processes transactions from a landing table and has read one million of them. Over the weekend, 300,000 more transactions land and get processed incorrectly by the broken deploy, and the checkpoint advances from transaction 1,000,000 to 1,300,000. Now try to repair it with the tools you have:
- Restore the tables to their Friday versions, and the incorrect data is removed, but the checkpoint still reads 1,300,000. When you resume your pipeline, it starts at transaction 1,300,001, so transactions 1,000,001 through 1,300,000 are never re-read. The weekend's data is now missing.
- Reset the checkpoint to Friday instead, and you get the opposite failure. The weekend's transactions are re-read and appended to the windows that already hold them, so every weekend payment is counted twice.
- Fix both at once, tables back to Friday, and the checkpoint reset to match, and a third thing could still go wrong. The gold aggregation could carry its own state like a watermark on Sunday night. Restoring the tables and offsets leaves that state untouched, so the replayed transactions arrive behind the watermark and get dropped as late.
Getting all three back to the same instant by hand, across every table and everything downstream of it, is the hard problem. So most teams do the only thing that is reliably safe: throw away the progress and reprocess from the beginning. While reprocessing from the beginning is foolproof, it is expensive and scales with your data. A three-day bug in a two-year-old pipeline costs two years of compute to fix.
What is SDP Rewind?
SDP Rewind moves all of this back together in one operation. You choose a point in time, and it rolls each table back to its version from then, resets every source position, and restores the operator state of any stateful flows, all to the same instant. Take that same Friday deploy, and one rewind removes all three problems:
- The tables and the checkpoint move back together. Rewind rolls both to Friday at once, so the re-read weekend transactions land in windows that no longer hold them. Nothing is skipped, so no data goes missing, and nothing is counted twice.
- Operator state moves back with them. Rewind also restores the gold aggregation's watermark to Friday, so the replayed rows arrive on time rather than behind it, and none are dropped late.
- Rewind Cascades to all downstream objects in the pipeline. It does not stop at one table and state. In a single operation, it rolls back all downstream Streaming Tables and Materialized Views, each with their own data, offsets and state, to the same Friday point.
Then you fix your code and start the pipeline the way you always do. There is no separate replay command. The pipeline resumes from the rewind point, reprocesses forward using the current definition, catches up to the present, and returns to normal incremental processing. The cost of the repair is proportional to the window you rewound, not to the pipeline's lifetime.
The points you can rewind to are called rewind points. The pipeline generates them automatically, about once an hour, and keeps them for 7 days. How far back you can actually go depends on your own retention: an aggressive VACUUM, or a short retention on a Kafka topic, can shorten the window before any product limit does.
In practice, this is a single API call. A rewind is issued against the pipeline with a rewind_spec that carries two things: a point in time to return to, and the datasets to rewind. The skeleton of the call looks like this:
databricks pipelines start-update <pipeline-id> --json '{
"cause": "API_CALL",
"rewind_spec": {
"rewind_timestamp": "<point-in-time>",
"datasets": [
{ "identifier": "<catalog>.<schema>.<table>", "cascade": true }
]
}
}'How it works under the hood
Rewind builds on Delta time travel. Delta already versions every table and can restore one to an earlier version, and that is the primitive Rewind uses to move each table's rows back. What Delta cannot do on its own is line up which table version goes with which source position, or coordinate that across a graph of tables. Rewind records for each table version, record the source position and operator state that produced it, then issue Delta's own RESTORE across the whole graph at once. The unit of recovery becomes the pipeline, not the table.
How a rewind actually runs
When you trigger a rewind, the pipeline walks its graph from the top down, works out the version and batch for each table, and checkpoints should return to, issues a Delta RESTORE on every table, and rewinds each checkpoint (and its operator state, where applicable) to the matching batch.
A Delta RESTORE does not delete the versions you roll back; it reapplies the old state as a new commit on top, so a table taken from version 100 to version 50 now sits at version 101 with version 50's contents. Left alone, the next run would re-read versions 51 through 100 and reprocess them, recreating the duplicate problem. Rewind hands the next run a set of skip ranges that tell it which now-phantom versions to read past, so the restart starts clean.
These steps are not a single atomic transaction, so a crash could leave the pipeline half-rewound. Two guards prevent that: RESTORE is idempotent, and the pipeline blocks ordinary updates until the rewind completes. Rewinding again to the same point converges, so a failed rewind leaves the pipeline stopped but recoverable.
Rewind in practice
We built a companion repository that stages the scenario above. It is a payments pipeline built on the medallion architecture: raw transactions land in bronze, get cleaned and converted in silver, and roll up in gold, which is a stateful five-minute windowed aggregation of settled dollars per merchant. It is small enough to read in a sitting and complete enough to break on purpose and recover for real. To follow along, clone it:
git clone https://github.com/databricks-solutions/sdp-rewind-replay
Before deploying, configure variables in databricks.yml to set the catalog and schema the tables land in, the landing table's name, and a warehouse ID (which the seeding script uses to maintain a steady rate of transaction seed). Then deploy it with Declarative Automation Bundles, which provisions the catalog, schema, landing table, pipeline, and dashboard in a single step using:
databricks bundle deploy
With the seeder feeding and the pipeline consuming, you have a live, healthy stream to break. The defect is a single wrong divisor in the silver transform, at src/pipeline.py:101. Amounts arrive as integer cents, so 7969 means $79.69, and the correct line divides by 100:
.withColumn("amount", F.col("amount_minor").cast("double") / 100)
# .withColumn("amount", F.col("amount_minor").cast("double") / 10)The two lines sit right below each other in the file, and exactly one is ever active. To introduce the bug, comment out the / 100 line, uncomment the / 10 line, and redeploy; to fix it later, swap them back.
This gets past both review and the platform. The value is still a double, so the schema is unchanged, and there is nothing for the pipeline to reject. The line still reads like an ordinary unit conversion. It is wrong in exactly the way that survives every automated check.
Nothing automated catches it, so the first place it surfaces is the dashboard that the bundle deploys. Settled dollars are plotted as bars, transaction count as a line, and normally they move together: more transactions, more money. Since the deployment, the bars spike while the line stays flat; the same payments at the same volume are valued incorrectly.
Payments Dashboard: Settled dollars jump ~10x right after the deploy, while transaction count stays flat
Recovery follows the three steps. First, deploy the corrected divisor, because replaying against the still-broken logic would only recreate the corruption. Then issue the rewind. Through the CLI, it is a single call, anchored against the UTC timestamp of the last known healthy state of the pipeline:
databricks pipelines start-update <pipeline-id> --json '{
"cause": "API_CALL",
"rewind_spec": {
"rewind_timestamp": "<timestamp>",
"datasets": [
{ "identifier": "catalog.schema.silver_payments", "cascade": true }
]
}
}'When the rewind finishes, every object from silver down has been carried back to the last healthy state, aligned to the same instant. From here, you start the pipeline. It picks up where the healthy history left off and processes forward through the corrected code. When it catches up, gold holds exactly the windows it would have held if the bad deploy had never happened: the corrupted rows replaced by correct ones, the healthy history before them untouched, no duplicated windows, no duplicated payments. Exactly-once semantics hold through a stateful aggregation, across a rewind, in a continuous pipeline.
Payments Dashboard: After the code fix and Rewind, settled dollars are back in line. Only the affected windows were reprocessed; the rest of the data was left untouched.
One declarative command returns the whole pipeline to a consistent state, reprocessing only the broken window and nothing else.
Try it
SDP Rewind is entering beta across Kafka, Delta, and streaming-table sources, with stateful operators. Refer to the documentation for more about the API. The full demo, including the pipeline, dashboard, and recovery flow, is available for deployment in your own workspace: databricks-solutions/sdp-rewind-replay. Run it, break it on purpose, and hit rewind yourself.
1 Comment
