https://community.databricks.com/t5/data-engineering/identify-the-associated-notenook-for-the-application-running/m-p/62661#M32036 <P><A href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.setJobDescription.html" target="_blank">https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.setJobDescription.html</A></P><P>spark.setJobDescription("my name") will make your life easier. Just put it in the notebook.</P><P>You should also put it after each action (show, count, toPandas, write, aggregation etc.)</P><P>&nbsp;</P> Tue, 05 Mar 2024 14:49:14 GMT Hubert-Dudek 2024-03-05T14:49:14Z https://community.databricks.com/t5/data-engineering/identify-the-associated-notenook-for-the-application-running/m-p/62605#M32009 <P>In spark UI, I can see the application running with the application ID, from this spark UI, could I able to see the which notebook is running with that applications is this possible?</P><P>I am interested in learning more about the jobs, stage how it works internally with spark with the help of spark UI in databricks, could you guide me(share the learning agenda or maerial or online course to check on it etc.. ) the path to learn more Indepth on it.</P><P>Note: as a first step I need to understand on how one complete notebook excution is taken care with respect to jobs and stage in the pyspark.&nbsp;&nbsp;</P><P>Thanks.</P> Mon, 04 Mar 2024 16:35:34 GMT https://community.databricks.com/t5/data-engineering/identify-the-associated-notenook-for-the-application-running/m-p/62605#M32009 Data_Engineer3 2024-03-04T16:35:34Z https://community.databricks.com/t5/data-engineering/identify-the-associated-notenook-for-the-application-running/m-p/62661#M32036 <P><A href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.setJobDescription.html" target="_blank">https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.setJobDescription.html</A></P><P>spark.setJobDescription("my name") will make your life easier. Just put it in the notebook.</P><P>You should also put it after each action (show, count, toPandas, write, aggregation etc.)</P><P>&nbsp;</P> Tue, 05 Mar 2024 14:49:14 GMT https://community.databricks.com/t5/data-engineering/identify-the-associated-notenook-for-the-application-running/m-p/62661#M32036 Hubert-Dudek 2024-03-05T14:49:14Z