I'd like to provide an update on this issue.

We have another DLT pipeline, which is a non-streaming (batch) scd type 1. This pipeline also failed with the same error message as indicated in my previous post. The data in this pipeline is completely different (source and structure) not a complex recursive struct, but just a flat record with some 50 static fields. So there is no chance that the schema has changed there. There is also no chance that key/sequence contains duplicates, as we have checked that carefully several times.

This pipeline worked correctly for the past 3 months. It failed yesterday. And failure in this pipeline also happened before, after which the pipeline worked again with us introducing no changes, but just re-starting it repeatedly.

This really looks erratic to me and I can't fathom any data related or logic related reasons that would be responsible for this which are under our control.

In order to gain understanding of the issue, I would request it again if you could provide some context in which the error message (in the original post) is generated. The content of the error message does not make sense to me and does not allow me to reason about possible causes.