- 3566 Views
- 5 replies
- 2 kudos
When running structured streaming jobs in production, what are the general best practices to reduce cost?
Consider a basic structured streaming use case of aggregating the data, perform some basic data cleaning transformation, and merge into a historical aggregate dataset.
- 3566 Views
- 5 replies
- 2 kudos
- 2 kudos
I second the recommendations: auto load with trigger, batch processing instead of continuous streaming where use case permits. In addition, test with a small batch firstfavor fewer larger workers over more smaller workersadjust your job cluster over...
- 2 kudos
- 2502 Views
- 3 replies
- 5 kudos
Resolved! What is the use case of having Azure Synapse(DWH) and Delta Lake ( Gold) given we can connect BI to delta directly
The curated zone is pushed to cloud data warehouse such as Synapse Dedicated SQL Pools which then acts as a serving layer for BI tools and analyst.I believe we can have models in gold layer and have BI connect to this layer or we can have serverless ...
- 2502 Views
- 3 replies
- 5 kudos
- 5 kudos
Thank you, so for a large workload, where we need lot of optimization we might need Synapse, but for a small/medium workload, we might have to stick to Delta Table
- 5 kudos