topic Re: Structured streaming in Databricks using delta table in Data Engineering

Structured streaming in Databricks using delta table

JissMathew — Tue, 19 Nov 2024 12:04:20 GMT

Hi everyone, I’m new to Databricks and exploring its features. I’m trying to implement Change Data Capture (CDC) from the bronze layer to the silver layer using streaming. Could anyone share sample code or reference materials for implementing CDC with streaming in Databricks? I’m also looking to better understand the concept of streaming in Databricks. Any guidance would be greatly appreciated 😇!!

Re: Structured streaming in Databricks using delta table

Walter_C — Tue, 19 Nov 2024 15:52:37 GMT

I will suggest you to go through blog https://www.databricks.com/blog/2022/04/25/simplifying-change-data-capture-with-databricks-delta-live-tables.html this will provide you with more details and few examples you can use

Re: Structured streaming in Databricks using delta table

JissMathew — Tue, 19 Nov 2024 18:03:00 GMT

Hi @Walter_C ,

I’m looking to implement streaming using Delta tables. While I understand that Delta Live Tables simplify this process, they are unfortunately not available to use in the free trial version. Could you help guide me on how to achieve streaming with Delta tables, or share any examples or resources for this approach? Thank you!

Re: Structured streaming in Databricks using delta table

Mike_Szklarczyk — Thu, 21 Nov 2024 12:11:51 GMT

Why you need to implement CDC from bronze to silver - that is strange.

Some time ago a kind person replied to me in a similar situation: 'Maybe you can more elaborate about your ground problem than asking about some solutian that you think is proper.' This is related to https://en.wikipedia.org/wiki/XY_problem

Do you need:

process your data from bronze to silver in the streaming manner (using Sructured Streaming)
process your data from bronze to silver using CDC (because in Bronze you have for example Delete operations on your data)
process tour data from bronze to silver using CDC in the streaming manner

Re: Structured streaming in Databricks using delta table

JissMathew — Thu, 21 Nov 2024 17:50:56 GMT

yeah , I need this case

process your data from bronze to silver in the streaming manner (using Sructured Streaming)

Re: Structured streaming in Databricks using delta table

Mike_Szklarczyk — Fri, 22 Nov 2024 10:42:27 GMT

Ok, so I recommend to familiar with this documents:
https://docs.databricks.com/en/structured-streaming/delta-lake.html#language-python
https://docs.databricks.com/en/structured-streaming/tutorial.html

Here you can find some sample generic transformation between batch and streaming approach:

# Batch approach: (spark.read .table("<table-name1>") .<some_transformations> .write .saveAsTable("<table-name3>") ) # Streaming approach: (spark.readStream .table("<table-name1>") .<some_transformations> .writeStream .trigger(availableNow=True) .option("checkpointLocation", "<checkpoint-path>") .saveAsTable("<table-name3>") )

Good luck 🙂

Re: Structured streaming in Databricks using delta table

Mike_Szklarczyk — Fri, 22 Nov 2024 10:44:06 GMT

You can also look at https://www.databricks.com/resources/demos#tutorials