Monitoring Databricks SQL Warehouse without Unity Catalog

[Dave_Nithio]

I am looking to monitor my SQL Warehouse, especially the 'Running Clusters' metric that is available in the monitoring tab of the warehouse. This shows the start and shut down time as well as the number of running clusters:

The issue I have run into is that SQL Warehouses do not seem to have the same types of logs that an all-purpose or cluster has. The documentation notes that the best option is to use System Tables to monitor your warehouse, including using the Warehouse Events table. The issue is that this requires the workspace to be enabled for Unity Catalog. My organization has not transitioned to Unity Catalog yet so these tables are not available. Is there an alternative method for getting access to this monitoring data that does not require unity catalog?

[Isi]

Hey @Dave_Nithio 

To monitor the “Running Clusters” metric for your SQL Warehouse, you can use the Databricks Cluster Events API. This API retrieves a list of events related to cluster activity, such as start and shutdown times, and provides paginated results if there are more events to retrieve.

Since Unity Catalog is not enabled in your workspace, the API offers an alternative to access the necessary monitoring data.

POST  /api/2.1/clusters/events

"Retrieves a list of events about the activity of a cluster. This API is paginated. If there are more events to read, the response includes all the parameters necessary to request the next page of events."

For more information, refer to the Databricks Cluster Events API documentation.

Hope this helps, 🙂

Isi