Jobs in Spark UI
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2025 05:12 AM
Is there a way to get the url where all the spark jobs which are created in a specific notebook run can be found? I am creating an audit framework, in that the requirement is to get the spark jobs of a specific task or a notebook run so that we can debug the job if there are any error.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2025 11:01 PM - edited 07-17-2025 11:02 PM
Hi @HariPrasad1, if I understand correctly, is it to programmatically retrieve the Spark UI URL (which has access to the list of jobs) in a notebook when a job is running?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2025 11:28 PM
Yes @eniwoke . You are correct.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2025 06:45 AM
Hi @HariPrasad1 here is a way to get the job list (note: works for non-serverless clusters)
from dbruntime.databricks_repl_context import get_context
cluster_id = spark.conf.get("spark.databricks.clusterUsageTags.clusterId")
workspaceUrl = spark.conf.get('spark.databricks.workspaceUrl')
context_id = spark.conf.get('spark.databricks.sparkContextId')
context__ = get_context()
workspaceId = context__.workspaceId
# For non serverless clusters
spark_job_ui_url = f"https://{workspaceUrl}/compute/sparkui/{cluster_id}/driver-{context_id}?o={workspaceId}"
print(spark_job_ui_url)kindly tell me if it works for you 🙂
