Pyspark streaming optimization we need to focus on

[YanhDong_68817]

What optimization we should focus on?

[Kumaran]

Hi @YanhDong_68817,

Thank you for posting question in the Databricks community.

I understand that you are looking for the documentation to optimize the pyspark streaming, please refer to this documentation for more help.

[Anonymous]

Hi @YanhDong_68817 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[Tharun-Kumar]

Hi @YanhDong_68817 

Could you let us know the source and destination of your streaming pipeline?

[Tharun-Kumar · Accepted solution]

@YanhDong_68817 
This document is one of the good places to start evaluating our streaming pipeline - https://docs.databricks.com/structured-streaming/production.html