Hello @DatabricksUser5 ,

You can’t override committed offsets in-place for a running DLT Kafka/Event Hubs stream. If a pipeline already has a checkpoint created, startingOffsets is ignored. To replay data, you must reset the streaming checkpoints or create a new checkpoint using FULL REFRESH in DLT, and the events must still be retained in Event Hubs.

For Kafka sources (including Event Hubs Kafka endpoint), Spark Structured Streaming:

• Uses startingOffsets only when the streaming query is initially created.
• After that, it always resumes from the offsets stored in the checkpoint directory, completely ignoring startingOffsets on restart.

You can refer to the KB for more details here: https://kb.databricks.com/streaming/offset-reprocessing-issues-in-streaming-queries-with-a-kafka-sou...

So, below are the mitigations for you scenario:

• Full refresh for the streaming table/pipeline (resets checkpoints and data) (OR)
• Reset streaming flow checkpoints (REST API, keeps table data) using https://docs.databricks.com/aws/en/ldp/updates#start-a-pipeline-update-to-clear-selective-streaming-...