topic Re: CDC / Event Driven Data Ingestion in Data Engineering

CDC / Event Driven Data Ingestion

Mey — Tue, 10 Feb 2026 04:58:01 GMT

Hello Guys,

I am planning to implement Event Driven Data Ingestion from Bronze -> Silver -> Gold layer in my project. Currently we are having batch processing approach for our data ingestion pipelines. We have decided to move away from batch process to Event Driven approach. Can some one guide me / throw some light on architectural design, steps / key factors I have to capture before design the CDC / Event Driven Scalable architecture. Also it would be good if you provide some examples or documents as a sample for my verification.

Thanks for all the helps so far!!!

Re: CDC / Event Driven Data Ingestion

saurabh18cs — Tue, 10 Feb 2026 10:08:22 GMT

Hi @Mey I would suggest you to use Lakeflow suite(lakeflow connect , lakeflow declaratve pipelines, lakeflow jobs) pipelines from databricks to achieve this event driven incremental workload.Below example is using SQL but you can also use python. read_files is an example of using databricks natively support autoloader for incremental streaming.

example workflow:

Re: CDC / Event Driven Data Ingestion

Advika — Wed, 11 Feb 2026 11:43:45 GMT

Hello @Mey,
This looks similar to your other post, where a solution was already accepted. To avoid confusion, let’s please continue the conversation in that thread.