topic Re: Jobs in Spark UI in Data Engineering

Jobs in Spark UI

HariPrasad1 — Thu, 17 Jul 2025 12:12:14 GMT

Is there a way to get the url where all the spark jobs which are created in a specific notebook run can be found? I am creating an audit framework, in that the requirement is to get the spark jobs of a specific task or a notebook run so that we can debug the job if there are any error.

Re: Jobs in Spark UI

eniwoke — Fri, 18 Jul 2025 06:02:13 GMT

Hi @HariPrasad1, if I understand correctly, is it to programmatically retrieve the Spark UI URL (which has access to the list of jobs) in a notebook when a job is running?

Re: Jobs in Spark UI

HariPrasad1 — Fri, 18 Jul 2025 06:28:55 GMT

Yes @eniwoke . You are correct.

Re: Jobs in Spark UI

eniwoke — Mon, 21 Jul 2025 13:45:59 GMT

Hi @HariPrasad1 here is a way to get the job list (note: works for non-serverless clusters)

from dbruntime.databricks_repl_context import get_context cluster_id = spark.conf.get("spark.databricks.clusterUsageTags.clusterId") workspaceUrl = spark.conf.get('spark.databricks.workspaceUrl') context_id = spark.conf.get('spark.databricks.sparkContextId') context__ = get_context() workspaceId = context__.workspaceId # For non serverless clusters spark_job_ui_url = f"https://{workspaceUrl}/compute/sparkui/{cluster_id}/driver-{context_id}?o={workspaceId}" print(spark_job_ui_url)

kindly tell me if it works for you 🙂