What are best practices for Spark streaming in Databricks
- is it good idea to consume multiple topics in one streaming job
- is Auto scaling recommended for spark streaming
- How many worker nodes we should choose for streaming job
- When should we run OPTIMIZE for continuously streaming topics
- any other things to consider to implement streaming jobs with high throughput
