cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Lake’s CDF Feature

THIAM_HUATTAN
Contributor III

https://www.databricks.com/notebooks/delta-lake-cdf.html

I am trying to understand the above article. Could someone explain to be the below questions?

a) From SELECT * FROM table_changes('gold_consensus_eps', 2)

why is consensus_eps values of 2.1 and 2.3 for date 3/1/2021? How are those values tabulated?

b) SELECT * FROM gold_consensus_eps

consensus_eps values of is now 2.3 for date 3/1/2021, is it because it gets from _change_type='update_postimage'?

1 ACCEPTED SOLUTION

Accepted Solutions

daniel_sahal
Honored Contributor III

@THIAM HUAT TAN​ 

update_preimage is a version of the row before update/merge and update_postimage is a version of row after the update/merge.

When doing SELECT * FROM table_changes it's showing both values - previous one and current one.

When doing SELECT * FROM tbl it's showing only the current value.

View solution in original post

6 REPLIES 6

daniel_sahal
Honored Contributor III

@THIAM HUAT TAN​ 

update_preimage is a version of the row before update/merge and update_postimage is a version of row after the update/merge.

When doing SELECT * FROM table_changes it's showing both values - previous one and current one.

When doing SELECT * FROM tbl it's showing only the current value.

thanks Daniel for your kind reply... yes, I understand that...

how does the consensus_eps values of 2.1 and 2.3 for date 3/1/2021 being tabulated/calculated?

I still have no idea how to calculate out those values above.

is that 2.1 = (2+2.2)/2 from the silver table?

how about that 2.3 value? how is it being derived?

daniel_sahal
Honored Contributor III

@THIAM HUAT TAN​ 

2.1 from previous + 2.4 from new = average of 2.25 ~2.3

oic, thanks a lot Daniel for your clarifications.

Anonymous
Not applicable

Hi @THIAM HUAT TAN​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.