topic Re: Submit new records from gold layer to postgres db in Data Engineering

Submit new records from gold layer to postgres db

jeremy98 — Sat, 28 Dec 2024 17:21:35 GMT

Hi community,
I want to ask you which is the best practice in your opinion to fill the data from gold layer to postgres db that is used to provide "real-time" data to an UI interface?

Thanks for any help!

Re: Submit new records from gold layer to postgres db

hari-prasad — Sun, 29 Dec 2024 07:37:21 GMT

Hello,

To begin, enable delta properties for your gold table if it isn’t already a delta table. Next, activate the Change Data Capture (CDC) delta property. This will give you a detailed log of insert, update, and delete operations at the row level, which you can use to perform DML operations on your Postgres tables with minimal coding effort.

Re: Submit new records from gold layer to postgres db

jeremy98 — Mon, 30 Dec 2024 13:31:25 GMT

Hello @hari-prasad,
Thanks you for your answer, considers that we are using DLT Pipelines, is it a good choice in this case? Because actually I don't see with DLT Pipelines the metadata for these materialized tables in this case these CDF statements