CDC / Event Driven Data Ingestion

[Mey]

Hello Guys,

I am planning to implement Event Driven Data Ingestion from Bronze -> Silver -> Gold layer in my project. Currently we are having batch processing approach for our data ingestion pipelines. We have decided to move away from batch process to Event Driven approach. Can some one guide me / throw some light on architectural design, steps / key factors I have to capture before design the CDC / Event Driven Scalable architecture. Also it would be good if you provide some examples or documents as a sample for my verification.

Thanks for all the helps so far!!!

[saurabh18cs · Accepted solution]

Hi @Mey I would suggest you to use Lakeflow suite(lakeflow connect , lakeflow declaratve pipelines, lakeflow jobs) pipelines from databricks to achieve this event driven incremental workload.Below example is using SQL but you can also use python. read_files is an example of using databricks natively support autoloader for incremental streaming.

example workflow:

[Advika · Accepted solution]

Hello @Mey,
This looks similar to your other post, where a solution was already accepted. To avoid confusion, let’s please continue the conversation in that thread.