- 1827 Views
- 2 replies
- 0 kudos
Lakehouse Architecture : How notebooks are organized and executed
We are implementing a lakehouse architecture and using Notebook to transform data from object storage. Most of the time our source is database for which there one folder per table in object storage.We have structure like below for various notebooksGO...
- 1827 Views
- 2 replies
- 0 kudos
- 0 kudos
Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with your request
- 0 kudos
- 661 Views
- 0 replies
- 0 kudos
Differently named storage accounts in different environments
Hi,I have a solution design question on which I am looking for some help. We have 2 environments in azure (dev and prod), each env has its own ADLS storage account with a different name of course. Within Databricks code we are NOT leveraging the mou...
- 661 Views
- 0 replies
- 0 kudos
- 1461 Views
- 1 replies
- 4 kudos
Resolved! What's the best architecture for Structured Streaming and why?
I am building an ETL pipeline which reads data from a Kafka topic ( data is serialized in Thrift format) and writes it to Delta Table in databricks. I want to have two layersBronze Layer -> which has raw Kafka dataSilver Layer -> which has deserializ...
- 1461 Views
- 1 replies
- 4 kudos
- 4 kudos
@John Constantine​ , "Bronze Layer -> which has raw Kafka data"If you use confluent.io, you can also utilize a direct sink to DataLake Storage - bronze layer."Silver Layer -> which has deserialized data"Then use Delta Live Tables to process it to del...
- 4 kudos