Getting data from the Spark query profiler

[IONA]

When you navigate to Compute > Select Cluster > Spark UI > JDBC/ODBC 

There you can see grids of Session stats and SQL stats. Is there any way to get this data in a query so that I can do some analysis?

Thanks

[BigRoux]

Hello Iona, You cannot natively query the exact Session stats and SQL stats from the JDBC/ODBC Spark UI via a simple SQL statement in Databricks today. However, advanced users and admins can access some of the underlying data via log tables (like prod.thrift_statements), Query History API, or specialized REST endpoints. For practical analysis, using the Query History API and parsing the results into Python or SQL for your analysis is the closest workaround currently available.

Hope this helps, Louis.

[WiliamRosa]

Hi @IONA,

Totally agree with @BigRoux  — to make his point actionable, here are official docs you can use:

Query history system table (system.query.history): https://docs.databricks.com/aws/en/admin/system-tables/query-history

Query History (overview/UI): https://docs.databricks.com/aws/en/sql/user/queries/query-history

Query History REST API: https://docs.databricks.com/api/workspace/queryhistory/list

Spark Thrift Server (why JDBC/ODBC UI grids aren’t exposed as tables; retention settings): https://spark.apache.org/docs/latest/configuration.html#spark-sql