topic What are Best Practices for Spark streaming in Databricks in Data Engineering https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24698#M17188 <P>What are best practices for Spark streaming in Databricks</P><OL><LI>is it good idea to consume multiple topics in one streaming job</LI><LI>is Auto scaling recommended for spark streaming</LI><LI>How many worker nodes we should choose for streaming job</LI><LI>When should we run OPTIMIZE for continuously streaming topics</LI><LI>any other things to consider to implement streaming jobs with high throughput </LI></OL> Mon, 14 Jun 2021 22:15:21 GMT Srikanth_Gupta_ 2021-06-14T22:15:21Z What are Best Practices for Spark streaming in Databricks https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24698#M17188 <P>What are best practices for Spark streaming in Databricks</P><OL><LI>is it good idea to consume multiple topics in one streaming job</LI><LI>is Auto scaling recommended for spark streaming</LI><LI>How many worker nodes we should choose for streaming job</LI><LI>When should we run OPTIMIZE for continuously streaming topics</LI><LI>any other things to consider to implement streaming jobs with high throughput </LI></OL> Mon, 14 Jun 2021 22:15:21 GMT https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24698#M17188 Srikanth_Gupta_ 2021-06-14T22:15:21Z Re: What are Best Practices for Spark streaming in Databricks https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24699#M17189 <P>What are best practices for Spark streaming in Databricks</P><OL><LI>is it good idea to consume multiple topics in one streaming job - Yes it is fine, we can create a fair pool and provide the infrastictrue to each stream so that it does not intervene between each other</LI><LI>is Auto scaling recommended for spark streaming - Nope </LI><LI>How many worker nodes we should choose for streaming job -Per partition one core </LI><LI>When should we run OPTIMIZE for continuously streaming topics - Any time </LI><LI>any other things to consider to implement streaming jobs with high throughput - Compute VM s are preferred as node </LI></OL><P></P> Fri, 18 Jun 2021 12:58:22 GMT https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24699#M17189 User16826994223 2021-06-18T12:58:22Z Re: What are Best Practices for Spark streaming in Databricks https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24700#M17190 <P>See our <A href="https://docs.databricks.com/spark/latest/structured-streaming/production.html" alt="https://docs.databricks.com/spark/latest/structured-streaming/production.html" target="_blank">docs</A> for other considerations when deploying a production streaming job.</P> Fri, 18 Jun 2021 17:37:30 GMT https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-spark-streaming-in-databricks/m-p/24700#M17190 craig_ng 2021-06-18T17:37:30Z