Hi,

We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.

Namely, we need to monitor the following:

• failed jobs : is a job failed
• table ingest rate : how much data is ingested
• table ingest lag : is a stream job further behind than expected
• table size : size of the current table being ingested into
• query runtime : the time a query has been running

Does anyone have any ideas on how to get those metrics (either through Prometheus or an alternative method)?