Hi @IONA ,

As @Louis_Frolio  correctly suggested there no native way to get stats from JDBC/ODBC Spark UI.

1. You can try to use query history system table, but it has limited number of metrics

%sql
SELECT *
FROM system.query.history

2. You can use /api/2.0/sql/history/queries endpoint with include_metrics flag enabled which should return to you following payload:

3. Metrics can be also obtained for following:

• Cluster metrics - you can export these with cluster logging. It's worth noting that ganglia is deprecated for newer runtimes
• Warehouse metrics - available through the API for query metrics
• Jobs performance - you can use the Jobs API 

4. And lastly, you can apache spark REST API monitoring endpoints which gives you access to multiple different metrics. Here, just for sake of an example I'm using it to get environment configuration of my cluster, but there are many many more metrics. Full list you can find at below location:

Monitoring and Instrumentation - Spark 4.0.0 Documentation

from dbruntime.databricks_repl_context import get_context
import requests

context = get_context()
host = context.browserHostName
cluster_id = context.clusterId


spark_ui_base_url = f"https://{host}/driver-proxy-api/o/0/{cluster_id}/40001/api/v1/"
endpoint = 'applications/local-1756797804565/environment'

response = requests.get(
    spark_ui_base_url + endpoint,
    headers={"Authorization": f"Bearer {context.apiToken}"}
)

if response.status_code == 200:
    try:
        data = response.json()
        print(data)
    except requests.exceptions.JSONDecodeError:
        print("Response is not valid JSON:")
        print(response.text)
else:
    print(f"Request failed with status code: {response.status_code}")
    print(f"Response: {response.text}")