Spark UI SQL/Dataframe tab missing queries

[kori73]

Hi,

Recently, I am having some problems viewing the query plans in the Spark UI SQL/Dataframe tab.

I would expect to see large query plans in the SQL tab where we can observe the details of the query such as the rows read/written/shuffled. However, I see just a single query with no details. The following screenshots might be helpful to understand the issue.

Here, I was expecting to see a large query plan (I can confirm this by looking at the Jobs tab).

 

I am having this issue in multiple Databricks Runtimes such as DBR 9.1 LTS, DBR 10.4 LTS , and DBR 12.2.

Any help is much appreciated!

[Anonymous]

@Koray Beyaz​ :

This issue may be related to a change in the default behavior of the Spark UI in recent versions of Databricks Runtime. In earlier versions, the Spark UI would display the full query plan for SQL and DataFrame operations in the SQL/Dataframe tab. However, in more recent versions, the default behavior has changed to show only a summary of the query plan.

To view the full query plan in the Spark UI, you can follow these steps:

1. Navigate to the SQL/Dataframe tab in the Spark UI.
2. Click on the operation that you want to view the query plan for. This will open a summary of the query plan.
3. In the summary view, you should see a button labeled "Show Additional Metrics". Click on this button to expand the view.
4. The expanded view will show additional details about the query plan, including the execution plan and the number of rows read/written/shuffled.

Alternatively, you can also view the query plan using the EXPLAIN command in SQL or the explain() method in DataFrame API. This will display the execution plan for the query in the notebook output, which you can analyze to understand the details of the query plan.

I hope this helps! Let me know if you have any further questions.