- 3678 Views
- 5 replies
- 2 kudos
When running structured streaming jobs in production, what are the general best practices to reduce cost?
Consider a basic structured streaming use case of aggregating the data, perform some basic data cleaning transformation, and merge into a historical aggregate dataset.
- 3678 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
I second the recommendations: auto load with trigger, batch processing instead of continuous streaming where use case permits. In addition, test with a small batch firstfavor fewer larger workers over more smaller workersadjust your job cluster over...
- 2 kudos
- 1393 Views
- 2 replies
- 3 kudos
Streaming Source for Feature Store (and outputMode)
To save computing resource and time, can I use streaming source in a batch mode (similar to Auto Loader) to update my feature store as my source table receives row update or is appended with new rows?
- 1393 Views
- 2 replies
- 3 kudos
Latest Reply
