raphaelblg
Databricks Employee
Databricks Employee

Hello @6502,

It appears you've used the `startingVersion` parameter in your streaming query, which causes the stream to begin processing data from the version prior to the DELETE operation version. However, the DELETE operation will still be processed in order, potentially resulting in failures.

To resolve this issue, consider the following options:

Roll back your table version to the version before the DELETE operation using time travel.

(https://docs.databricks.com/en/delta/history.html#restore-a-delta-table-to-an-earlier-state)

or


2. Add the `ignoreDeletes` or `skipChangeCommits` parameter to your query. You can find more information on this in the Databricks documentation.

(https://docs.databricks.com/en/structured-streaming/delta-lake.html#ignore-updates-and-deletes)

 

Should you have any questions or concerns, please don't hesitate to respond to this message. I'm here to help!

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks