topic Re: Delta Lake’s CDF Feature in Data Engineering

Delta Lake’s CDF Feature

THIAM_HUATTAN — Mon, 15 May 2023 12:50:55 GMT

https://www.databricks.com/notebooks/delta-lake-cdf.html

I am trying to understand the above article. Could someone explain to be the below questions?

a) From SELECT * FROM table_changes('gold_consensus_eps', 2)

why is consensus_eps values of 2.1 and 2.3 for date 3/1/2021? How are those values tabulated?

b) SELECT * FROM gold_consensus_eps

consensus_eps values of is now 2.3 for date 3/1/2021, is it because it gets from _change_type='update_postimage'?

Re: Delta Lake’s CDF Feature

daniel_sahal — Tue, 16 May 2023 05:46:01 GMT

@THIAM HUAT TAN

update_preimage is a version of the row before update/merge and update_postimage is a version of row after the update/merge.

When doing SELECT * FROM table_changes it's showing both values - previous one and current one.

When doing SELECT * FROM tbl it's showing only the current value.

Re: Delta Lake’s CDF Feature

THIAM_HUATTAN — Tue, 16 May 2023 06:07:32 GMT

thanks Daniel for your kind reply... yes, I understand that...

how does the consensus_eps values of 2.1 and 2.3 for date 3/1/2021 being tabulated/calculated?

I still have no idea how to calculate out those values above.

Re: Delta Lake’s CDF Feature

THIAM_HUATTAN — Tue, 16 May 2023 06:13:57 GMT

is that 2.1 = (2+2.2)/2 from the silver table?

how about that 2.3 value? how is it being derived?

Re: Delta Lake’s CDF Feature

daniel_sahal — Tue, 16 May 2023 06:42:30 GMT

@THIAM HUAT TAN

2.1 from previous + 2.4 from new = average of 2.25 ~2.3

Re: Delta Lake’s CDF Feature

THIAM_HUATTAN — Tue, 16 May 2023 06:48:27 GMT

oic, thanks a lot Daniel for your clarifications.

Re: Delta Lake’s CDF Feature

Anonymous — Tue, 23 May 2023 08:58:26 GMT

Hi @THIAM HUAT TAN

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!