topic Structured Streaming schemaTrackingLocation does not work with starting_version in Data Engineering

Structured Streaming schemaTrackingLocation does not work with starting_version

Volker — Wed, 16 Oct 2024 13:54:36 GMT

Hello Community,

I came across a strange behviour when using structured streaming on top of a delta table.

I have a stream that I wanted to start from a specific version of a delta table using the option option(

"starting_version", x) because I did not want to stream all the data of source the table but only the newly arriving one. To accomodate future (non-additive) schema changes I also set the option option("schemaTrackingLocation", checkpoint_location).

Now, if I change the schema of the source table the DataStreamReader does not pick up the schema changes and writes these to the schemaTrackingLocation but still infers the old schema and I can't get it to pick up the schema changes.

After some trial and error I found out that the starting_version is probably the cause of the issue since I tried changing the schema on a stream without setting the starting_version option and it worked as intended and could pick up the schema changes on the source table.

I'm a bit confused since the starting_version should only have an effect when starting the stream and otherwise be ignored, as from the docs:

They take effect only when starting a new streaming query. If a streaming query has started and the progress has been recorded in its checkpoint, these options are ignored. https://docs.databricks.com/en/structured-streaming/delta-lake.html#specify-initial-position

Did anybody have a similar problem? Is this an intended behaviour? How can I solve this issue? Where could I raise this issue?

Re: Structured Streaming schemaTrackingLocation does not work with starting_version

Volker — Fri, 18 Oct 2024 15:03:58 GMT

I found that it actually is not related to specifying the starting_version.

I think I found the flaw in the flow how the schema is updated in the schemaTrackingLocation:

On the first readStream operation the _schema_log_... gets created
On the first writeStream operation the schema gets written to the _schema_log_
readStream will now read in the source table with the schema from _schema_log_
the schema in _schema_log_ only gets updated on a writeStream operation if there is a schema change detected in the source table
The check if the source table schema was updated happens after checking if the data schema and the target schema are compatible
If the source table schema and target table schema get updated simultaneously then the stream fails since it detects a schema mismatch between the data schema (which is the original schema) and the target table schema
Because the stream fails at this point the schema in the _schema_log_ does not get updated as well and the readStream will always only read in the original schema of the source table even though the schema changed.

This is quite annoying behaviour because of this you would first need to adapt the schema of the source table, let the stream fail since it detected a schema change (which will cause an update of the schema in the _schema_log_) and then update the schema of the target table.
I know that I could use schema evolution but I do not want to use it if possible.
Does anybody have experience with this and has a workaround?

Re: Structured Streaming schemaTrackingLocation does not work with starting_version

mark_ott — Mon, 17 Nov 2025 12:04:14 GMT

This issue is related to how Delta Lake’s structured streaming interacts with schema evolution and options like startingVersion and schemaTrackingLocation. The behavior you've observed has been noted by other users, and can be subtle due to how checkpointing, versioning, and schema tracking are handled in combination. Here’s a breakdown, with solutions:

Core Issue

Setting startingVersion as an option in your stream appears to interfere with schema evolution, resulting in the stream persisting the old schema—even after the underlying Delta table’s schema has changed and updates are written to your schemaTrackingLocation.

When you remove startingVersion, the DataStreamReader detects schema changes correctly, provided schema tracking is enabled. From Databricks documentation, startingVersion should only be relevant for initializing a new stream, not for resumes from an existing checkpoint.

Why Does This Happen?

Schema Tracking and startingVersion:
- When startingVersion is set, it can impact which version of the table the streaming query starts reading from—even if a checkpoint exists. Certain system versions and Spark releases may not fully disregard this option after checkpoint initialization due to nuanced implementation details behind the scenes.
- The schema stored at schemaTrackingLocation is used for schema management, but if the stream is “stuck” at an older version due to how startingVersion is interpreted, it may not trigger schema updates.
Checkpoints and Restart Behavior:
- On restart, if a checkpoint exists, the stream should ignore startingVersion. However, if the checkpoint is missing or corrupted, or if the startingVersion option is reapplied incorrectly, the schema may not evolve as expected.

Workaround (if you need to retain startingVersion logic):

Start your stream without the startingVersion once the checkpoint is established, so ongoing runs see schema changes.
For testing, you can clear out your checkpoint directory (careful: this resets your offsets and may replay data) then set startingVersion to reinitialize from that version, but be sure to understand the replay implications.

References

Summary

This is not fully “intended” behavior, but more a side-effect of how options and checkpointing interact in specific tool versions. Removing startingVersion after initial setup, maintaining your checkpoint, and enabling schema tracking is the correct pattern for evolving schemas in Delta Lake structured streaming. If the problem persists after following this approach and upgrading, it may warrant a support ticket or GitHub issue.

topic Structured Streaming schemaTrackingLocation does not work with starting_version in Data Engineering

Structured Streaming schemaTrackingLocation does not work with starting_version

Re: Structured Streaming schemaTrackingLocation does not work with starting_version

Re: Structured Streaming schemaTrackingLocation does not work with starting_version

Core Issue

Why Does This Happen?

Suggested Solution

Workaround (if you need to retain startingVersion logic):

References

Summary