cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

How to retrieve a Job Name from the SparkContext

HHol
New Contributor

We are currently starting to build certain data pipelines using Databricks.

For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.

We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the SparkContext using:

 

from pyspark.context import SparkContext
sc = SparkContext.getOrCreate()
sc.getLocalProperty("spark.databricks.job.id")
sc.getLocalProperty("spark.databricks.job.parentRunId")
sc.getLocalProperty("spark.databricks.job.runId")
 
Now we would also like to get the name of the current Job which is running the wheel. Is there any local property containing that information? So far we did not manage to retrieve all available Keys which are stored as local Properties. Is there any documentation describing all available Properties? We did not find anything out there...
 
Thanks in advance for your advice!
0 REPLIES 0
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.