What are the best practices for implementing non-stop streaming in a Medallion Architecture with a Star Schema?
Use Case:
We have operational data and need to enable near real-time reporting in Power BI, with a maximum latency of 3 minutes. No Delta live tables.
Key Questions:
How should we curate dimensions and facts when transitioning data from Silver to Gold using Structured Streaming?
Could you provide examples or proven approaches for fact-dimension joins in a streaming context?
How can we use CDC in here?
In case of more questions and clarification happy to answer your questions