topic How to deal with Slow Jobs? in Community Articles https://community.databricks.com/t5/community-articles/how-to-deal-with-slow-jobs/m-p/71357#M105 <P>Definitely configure <A href="https://docs.databricks.com/en/workflows/jobs/job-notifications.html#configure-notifications-for-slow-running-or-late-jobs" target="_self">job timeouts,</A>&nbsp;and configure&nbsp;<A href="https://docs.databricks.com/en/workflows/jobs/job-notifications.html#add-email-and-system-notifications-for-job-events" target="_self">notifications.</A></P> <P>This will help you to identify slowness due to various factors.</P> <P>It is crucial to also investigate and fix the issue causing the slowness.</P> <OL> <LI>The first step is to identify the problem. This can be done by <STRONG>comparing the run</STRONG> times of the same job at different instances.&nbsp;</LI> <LI>&nbsp;The next step is to analyze the details of the job. This can be done by <STRONG>checking the SQL query plan</STRONG>, the read time, and the cloud storage request duration, etc from the Spark UI.</LI> <LI>External factors such as <STRONG>storage</STRONG> and <STRONG>network</STRONG> can also affect the job run time.&nbsp;<STRONG>Checking Logs</STRONG>&nbsp;and some system level commands can give insights if the vm is still up and running.</LI> </OL> <P>While comparing the two runs (good and bad), try answering the following</P> <UL> <LI>Is this an intermittent failure or it degraded after a certain point of time.</LI> <LI>If it is slow after a certain time consistently was there a DBR change?</LI> <LI>Is the volume of the data the same?</LI> <LI>Have the cluster configs changed/updated?</LI> </UL> <P>While comparing the DAGs and sql plan</P> <UL> <LI>Look for the stages that took most time.</LI> <LI>Use filters and reduce data size.</LI> <LI>Check the joins, metrics.</LI> </UL> <P>In the logs</P> <UL> <LI>You can check for errors, warnings.</LI> <LI>Go to the timestamp when the stage was delayed.</LI> <LI>Compare it with a run where it took the expected time.</LI> </UL> <P>Bonus tip -&nbsp;<SPAN>Enable speculative execution for tasks to re-run slow tasks in parallel.&nbsp;spark.speculation=<SPAN class="hljs-literal">true</SPAN></SPAN></P> <P><SPAN>Keep a look out for my new post for more tuning on a slow task/jobs.</SPAN></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> Sun, 02 Jun 2024 06:04:10 GMT NandiniN 2024-06-02T06:04:10Z How to deal with Slow Jobs? https://community.databricks.com/t5/community-articles/how-to-deal-with-slow-jobs/m-p/71357#M105 <P>Definitely configure <A href="https://docs.databricks.com/en/workflows/jobs/job-notifications.html#configure-notifications-for-slow-running-or-late-jobs" target="_self">job timeouts,</A>&nbsp;and configure&nbsp;<A href="https://docs.databricks.com/en/workflows/jobs/job-notifications.html#add-email-and-system-notifications-for-job-events" target="_self">notifications.</A></P> <P>This will help you to identify slowness due to various factors.</P> <P>It is crucial to also investigate and fix the issue causing the slowness.</P> <OL> <LI>The first step is to identify the problem. This can be done by <STRONG>comparing the run</STRONG> times of the same job at different instances.&nbsp;</LI> <LI>&nbsp;The next step is to analyze the details of the job. This can be done by <STRONG>checking the SQL query plan</STRONG>, the read time, and the cloud storage request duration, etc from the Spark UI.</LI> <LI>External factors such as <STRONG>storage</STRONG> and <STRONG>network</STRONG> can also affect the job run time.&nbsp;<STRONG>Checking Logs</STRONG>&nbsp;and some system level commands can give insights if the vm is still up and running.</LI> </OL> <P>While comparing the two runs (good and bad), try answering the following</P> <UL> <LI>Is this an intermittent failure or it degraded after a certain point of time.</LI> <LI>If it is slow after a certain time consistently was there a DBR change?</LI> <LI>Is the volume of the data the same?</LI> <LI>Have the cluster configs changed/updated?</LI> </UL> <P>While comparing the DAGs and sql plan</P> <UL> <LI>Look for the stages that took most time.</LI> <LI>Use filters and reduce data size.</LI> <LI>Check the joins, metrics.</LI> </UL> <P>In the logs</P> <UL> <LI>You can check for errors, warnings.</LI> <LI>Go to the timestamp when the stage was delayed.</LI> <LI>Compare it with a run where it took the expected time.</LI> </UL> <P>Bonus tip -&nbsp;<SPAN>Enable speculative execution for tasks to re-run slow tasks in parallel.&nbsp;spark.speculation=<SPAN class="hljs-literal">true</SPAN></SPAN></P> <P><SPAN>Keep a look out for my new post for more tuning on a slow task/jobs.</SPAN></P> <P>&nbsp;</P> <P>&nbsp;</P> <P>&nbsp;</P> Sun, 02 Jun 2024 06:04:10 GMT https://community.databricks.com/t5/community-articles/how-to-deal-with-slow-jobs/m-p/71357#M105 NandiniN 2024-06-02T06:04:10Z