topic Re: Spark Jobs View on a serverless cluster!! Naah..... in Community Articles

Spark Jobs View on a serverless cluster!! Naah.....

Raman_Unifeye — Thu, 04 Dec 2025 22:41:21 GMT

Have you ever noticed (and wondered) that the wonderful Spark Job UI is no longer available in the databricks notebook if the cell is executed using 'serverless' cluster?

Tradionally, whenever we run the spark code (action command), we used to see the Job Runs (job 1, Job2 etc) which navigate to the Spark famous UI with DAG and all run details. That view is not available if your notebook is attached to interactive serverless cluster.

Why:

Serverless abstraction: In serverless interactive clusters (and SQL Warehouses), Databricks fully manages the Spark driver and executors behind the scenes. You don’t get direct visibility into the JVM processes or Spark UI because those resources are ephemeral and multi‑tenant.
No dedicated driver node: The Spark driver that normally hosts the Spark UI (with DAG, stages, tasks) isn’t exposed in serverless mode. Databricks hides it to enforce isolation and simplify operations.
Security & multi‑tenancy: The Spark UI can reveal low‑level details about cluster internals. In shared/serverless environments, exposing that could leak information across tenants, so Databricks disables it.

Monitoring alternative

Instead of the Spark UI, Databricks provides query history, execution plans (EXPLAIN), and job run details in the Databricks workspace. These give you visibility into performance without exposing the raw DAG view.

Query History (SQL Warehouses): Shows query execution times, resource usage, and status.
Job Run Details (Jobs UI): For scheduled pipelines, you can see task durations, logs, and outcomes.
df.explain(True): Prints the physical plan, which is the closest to DAG inspection in serverless mode.
Metrics in Databricks UI: Cluster metrics, query profiles, and Delta Live Tables monitoring dashboards.

Happy serverless!!! 🙂

Re: Spark Jobs View on a serverless cluster!! Naah.....

bianca_unifeye — Fri, 05 Dec 2025 12:59:21 GMT

Very good observation, Raman! Thank you for bring this to the community attention!

Re: Spark Jobs View on a serverless cluster!! Naah.....

Senga98 — Mon, 08 Dec 2025 06:15:38 GMT

Hi Raman

Thank you for the amazing insights! I am trying to understand more about SQL Warehouses - is it managed by Unity Catalog? From what I could gather, SQL Warehouse is a compute layer, not a data layer and therefore not managed by Unity Catalog. Is this right?

Re: Spark Jobs View on a serverless cluster!! Naah.....

Raman_Unifeye — Mon, 08 Dec 2025 09:50:36 GMT

@Senga98 - I see you already had separate thread on your question and it is already answered by community 🙂

Re: Spark Jobs View on a serverless cluster!! Naah.....

Senga98 — Mon, 08 Dec 2025 10:01:43 GMT

Thanks, @Raman_Unifeye! Yes, I noticed the answer came in shortly after I posted this comment. All good now 😊