topic Re: Can I have sequence guarantee when replicate with CDF in Data Engineering

Can I have sequence guarantee when replicate with CDF

MikeGo — Fri, 08 Nov 2024 08:29:33 GMT

Hi team,

I have a delta table src, and somehow I want to replicate it to another table tgt with CDF, sort of

(spark .readStream .format("delta") .option("readChangeFeed", "true") .table('src') .writeStream .format("delta") .outputMode("append") .option("checkpointLocation", 'xxx') .toTable('tgt'))

Thanks to CDF, in tgt table I can have _commit_version. Can I have guarantee the _commit_version shows in the right sequence as they are in src table?

Thanks

Re: Can I have sequence guarantee when replicate with CDF

Mounika_Tarigop — Wed, 13 Nov 2024 20:47:04 GMT

The _commit_version is a part of the Delta Lake transaction log and is committed at the same time as the new data. This means that the changes are processed in the order they were committed in the source table.Ensure that CDF is enabled on your source Delta table (src). This allows you to capture changes (inserts, updates, deletes) in the source table.

The code snippet you provided, which correctly sets up the streaming read and write operations. By following this, you can ensure that the _commit_version in the target table (tgt) reflects the correct sequence of changes as they occurred in the source table (src). This guarantees that the data in the target table is consistent with the source table in terms of the order of commits

Re: Can I have sequence guarantee when replicate with CDF

MikeGo — Wed, 13 Nov 2024 22:54:56 GMT

Thanks. If the replicated table can have the _commit_version in strict sequence, I can take it as a global ever-incremental col and consume the delta of it (e.g. in batch way) with

select * from replicated_tgt where _commit_version > ( selecct last_version_offset = max(_commit_version) from downstream )

Thanks.