Hubert-Dudek
Databricks MVP

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.setJobDescription....

spark.setJobDescription("my name") will make your life easier. Just put it in the notebook.

You should also put it after each action (show, count, toPandas, write, aggregation etc.)

 


My blog: https://databrickster.medium.com/