topic Re: CDC DLT in Data Engineering

CDC DLT

Stellar — Wed, 06 Mar 2024 08:39:37 GMT

Hi all,

I would appreciate some clarity regarding the DLT and CDC. So my first question would be, when it comes to the "source" table in the synta, is that CDC table or? Further, if we want to use only databricks, would mounting foreign catalog be a good thing? Also, what would be some of the solutions you guys implemented to initially pull in the data? I need to pull in certain portion of the data not the whole table and then continue with DLT CDC.

Also, if my CDC table is in database(SQL Server), what would be the preferred method to connect to that database and read cdc table from Databricks?

Thanks!

#CDC #Databricks

Re: CDC DLT

Stellar — Thu, 07 Mar 2024 11:59:16 GMT

@Retired_mod Thanks for the explanations! What I am not sure about is the following. I create streaming table within the apply changes syntax. However I need to have a table already with the data. Does this mean that in the apply changes, create streaming table syntax is just refrencing existing table as delta live table or? Further, do we always load the whole cdc table? Sorry if my questions are not good but I am trying to figure this out. Also just to confirm my understanding these keys are not from the cdc table but rather from the table we wish to apply changes to because we need to connect two records together.