Delta Lake’s CDF Feature

[THIAM_HUATTAN]

https://www.databricks.com/notebooks/delta-lake-cdf.html

I am trying to understand the above article. Could someone explain to be the below questions?

a) From SELECT * FROM table_changes('gold_consensus_eps', 2)

why is consensus_eps values of 2.1 and 2.3 for date 3/1/2021? How are those values tabulated?

b) SELECT * FROM gold_consensus_eps

consensus_eps values of is now 2.3 for date 3/1/2021, is it because it gets from _change_type='update_postimage'?

[daniel_sahal · Accepted solution]

@THIAM HUAT TAN​ 

update_preimage is a version of the row before update/merge and update_postimage is a version of row after the update/merge.

When doing SELECT * FROM table_changes it's showing both values - previous one and current one.

When doing SELECT * FROM tbl it's showing only the current value.

[THIAM_HUATTAN]

thanks Daniel for your kind reply... yes, I understand that...

how does the consensus_eps values of 2.1 and 2.3 for date 3/1/2021 being tabulated/calculated?

I still have no idea how to calculate out those values above.

[THIAM_HUATTAN]

is that 2.1 = (2+2.2)/2 from the silver table?

how about that 2.3 value? how is it being derived?

[daniel_sahal]

@THIAM HUAT TAN​ 

2.1 from previous + 2.4 from new = average of 2.25 ~2.3

[THIAM_HUATTAN]

oic, thanks a lot Daniel for your clarifications.

[Anonymous]

Hi @THIAM HUAT TAN​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

.