I'm trying to accomplish a straightforward scd-type1 on a table. As such, the table has key and sequence number. There are a lot of other data elements, but I think these are just along for a ride - as this is the objective of scd type 1 - to capture the latest update.

As I said I did verify that each key/sequence number combination is good, that is there are multiple sequence numbers per key and they are not repeating, the sequence numbers are all sequential. The objective is to pick up the latest update as per highest sequence number.

Now, when I consider this operation, it appears consistent and reliable logically, but the issue is that it seems to fail sometimes on DLT pipeline.

It is possible that I'm missing or not understanding something.

It is possible that this is not the source of the error, but the issue is that this is what I'm trying to do and it fails.

But can you verify that my logic that I outlined above is correct.

If it is, then it is all that is necessary for me to understand to implement SCD type 1 in DLT.

Please let me know what I'm missing something

I'll be happy to walk you through the actual pipeline but it would require an online meeting.

The pipeline is simple consisting of several steps (above).

As per my original post error message, if you could clarify what it means, when it happens, what it complains about, I would greatly appreciate it. At the moment I can't imagine what the problem might be. Without understanding the problem, it is hard to find a solution.