topic Re: Seeking Practical Example for Structured Streaming with Delta Tables in Medallion Architecture in Administration & Architecture

Seeking Practical Example for Structured Streaming with Delta Tables in Medallion Architecture

JissMathew — Fri, 29 Nov 2024 08:33:36 GMT

Hi everyone,

I’m working on implementing Structured Streaming in Databricks to capture Change Data Capture (CDC) as part of a Medallion Architecture (Bronze, Silver, and Gold layers). While Microsoft’s documentation provides a theoretical approach, I’m looking for hands-on examples or code snippets that you’ve successfully used in a real-world project.

Specifically, I’d like to understand:

How to ingest data into a Delta table (Bronze layer) using Auto Loader or another streaming method.
How to process this data incrementally to create CDC and propagate changes to Silver and Gold layers.
Any recommendations for configurations or optimizations to manage schema evolution and large datasets effectively.

If anyone has experience with this and can share practical examples or insights beyond the documentation, it would be greatly appreciated!

Thank you in advance!

Re: Seeking Practical Example for Structured Streaming with Delta Tables in Medallion Architecture

szymon_dybczak — Fri, 29 Nov 2024 12:31:43 GMT

Hi @JissMathew ,

Do you have access to databricks academy? I believe in their data engineering track there's pleny of example notebooks.
Or you can try dbdemos. For example, here you can find demo notebook for autoloader

Databricks Autoloader (cloudfile)

If you'd like to test it on your databricks instance just do the following:

%pip install dbdemos

import dbdemosdbdemos.install('auto-loader')

For CDC pipeline you can use following:

CDC Pipeline With Delta | Databricks

Re: Seeking Practical Example for Structured Streaming with Delta Tables in Medallion Architecture

JissMathew — Wed, 04 Dec 2024 06:33:00 GMT

Hi @szymon_dybczak , Thank you very much. Your reply provided me with an excellent reference solution. I had been struggling with structured streaming, and your help was incredibly valuable and insightful.