How to view the SQL Query History of traditional Databricks cluster (not Databricks SQL)?

[jwilliam]

I tried use the Spark Cluster UI. But the queries are truncated.

[Anonymous · Accepted solution]

The traditional cluster doesn't save a list of SQL queries. It has a SQL UI, but it shows a dag of what happened rather than the full query.

[jwilliam]

Hi @Joseph Kambourakis​ , does Databricks have any plans to support this in the future?

[BilalAslamDbrx]

@John William​ it's definitely something we'd like to do.

[walkermaster12]

In Apache Spark prior to 2.1, once a SQL query was run, there was no way to re-run it; all history was lost. Spark SQL introduced the "replay" functionality in Spark 2.1.0, enabling users to re-run any query they have already run. You can run a query once, save the query plan, and then later open the saved plan and run the query again. This type of history is stored in a table named sql query plan in the database. This can be seen by using the SHOW TABLES statement in the Spark SQL console.