https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/m-p/48959#M28434 <P>I find it quite hard to understand Spark UI for my pyspark pipelines. For example, when one writes `spark.read.table("sometable").show()` it shows:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="igorgatis_0-1697034219608.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/4365iA4E6F5BAC187DA1C/image-size/large/is-moderation-mode/true?v=v2&amp;px=999" role="button" title="igorgatis_0-1697034219608.png" alt="igorgatis_0-1697034219608.png" /></span></P><P>I learned that `DataFrame` API actually may spawn jobs before running the actual job. In the example above, job 15 collects data which is used in job 16. In both cases, the description gives no clue on what is going on.<BR /><BR /></P><P>Clicking on job 15 link, it shows a stage that looks like this:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="igorgatis_1-1697034492125.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/4366i54D3BE2E08668356/image-size/large/is-moderation-mode/true?v=v2&amp;px=999" role="button" title="igorgatis_1-1697034492125.png" alt="igorgatis_1-1697034492125.png" /></span></P><P>Whose link leads to:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="igorgatis_2-1697034528335.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/4367i8E8BD3912BECDBE6/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="igorgatis_2-1697034528335.png" alt="igorgatis_2-1697034528335.png" /></span></P><P>The job 16 is quite similar though it mentions the table name. Things get messier when DAG gets more complex.</P><P>Is there a recommended way to improve this? I'm aware of `<SPAN>setJobDescription</SPAN><SPAN>`, `setLocalProperty` (with `callSite.short` and `callSite.long` but dealing with them directly is also not easy.</SPAN></P><P>&nbsp;</P> Wed, 11 Oct 2023 14:32:57 GMT igorgatis 2023-10-11T14:32:57Z https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/m-p/48959#M28434 <P>I find it quite hard to understand Spark UI for my pyspark pipelines. For example, when one writes `spark.read.table("sometable").show()` it shows:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="igorgatis_0-1697034219608.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/4365iA4E6F5BAC187DA1C/image-size/large/is-moderation-mode/true?v=v2&amp;px=999" role="button" title="igorgatis_0-1697034219608.png" alt="igorgatis_0-1697034219608.png" /></span></P><P>I learned that `DataFrame` API actually may spawn jobs before running the actual job. In the example above, job 15 collects data which is used in job 16. In both cases, the description gives no clue on what is going on.<BR /><BR /></P><P>Clicking on job 15 link, it shows a stage that looks like this:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="igorgatis_1-1697034492125.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/4366i54D3BE2E08668356/image-size/large/is-moderation-mode/true?v=v2&amp;px=999" role="button" title="igorgatis_1-1697034492125.png" alt="igorgatis_1-1697034492125.png" /></span></P><P>Whose link leads to:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="igorgatis_2-1697034528335.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/4367i8E8BD3912BECDBE6/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="igorgatis_2-1697034528335.png" alt="igorgatis_2-1697034528335.png" /></span></P><P>The job 16 is quite similar though it mentions the table name. Things get messier when DAG gets more complex.</P><P>Is there a recommended way to improve this? I'm aware of `<SPAN>setJobDescription</SPAN><SPAN>`, `setLocalProperty` (with `callSite.short` and `callSite.long` but dealing with them directly is also not easy.</SPAN></P><P>&nbsp;</P> Wed, 11 Oct 2023 14:32:57 GMT https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/m-p/48959#M28434 igorgatis 2023-10-11T14:32:57Z https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/m-p/49927#M28652 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/91285">@igorgatis</a>,</P> <P>A polite reminder. Have you had a chance to review my colleague's reply? Please inform us if it contributes to resolving your query.</P> Thu, 26 Oct 2023 16:24:06 GMT https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/m-p/49927#M28652 jose_gonzalez 2023-10-26T16:24:06Z