cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Identify the associated notenook for the application running from the spark UI

Data_Engineer3
Contributor III

In spark UI, I can see the application running with the application ID, from this spark UI, could I able to see the which notebook is running with that applications is this possible?

I am interested in learning more about the jobs, stage how it works internally with spark with the help of spark UI in databricks, could you guide me(share the learning agenda or maerial or online course to check on it etc.. ) the path to learn more Indepth on it.

Note: as a first step I need to understand on how one complete notebook excution is taken care with respect to jobs and stage in the pyspark.  

Thanks.

1 REPLY 1

Hubert-Dudek
Esteemed Contributor III

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.SparkContext.setJobDescription....

spark.setJobDescription("my name") will make your life easier. Just put it in the notebook.

You should also put it after each action (show, count, toPandas, write, aggregation etc.)

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group