What's the best architecture for Structured Streaming and why?

Constantine — Wed, 06 Apr 2022 22:47:43 GMT

I am building an ETL pipeline which reads data from a Kafka topic ( data is serialized in Thrift format) and writes it to Delta Table in databricks. I want to have two layers

Bronze Layer -> which has raw Kafka data

Silver Layer -> which has deserialized data

I can think of two ways to do it

First way is to read data from Kafka, write the raw data to bronze then read data from bronze and decode it and write it to silver

Second way is to read data from Kafka, write data to bronze and simultaneously decode the data and write it to silver.

I am trying to understand the advantages & disadvantages of each solution. Solution two is much easier to implement but feels like solution one is more fault tolerant

Re: What's the best architecture for Structured Streaming and why?

Hubert-Dudek — Thu, 07 Apr 2022 10:15:43 GMT

@John Constantine , "Bronze Layer -> which has raw Kafka data"

If you use confluent.io, you can also utilize a direct sink to DataLake Storage - bronze layer.

"Silver Layer -> which has deserialized data"

Then use Delta Live Tables to process it to delta silver. (file notification mode recommended)

topic Re: What's the best architecture for Structured Streaming and why? in Data Engineering

What's the best architecture for Structured Streaming and why?

Re: What's the best architecture for Structured Streaming and why?