https://community.databricks.com/t5/get-started-discussions/monitoring-job-metrics/m-p/42957#M854 <P>I have reposted this post in "Administation and Architecture"</P><P><A href="https://community.databricks.com/t5/administration-architecture/monitoring-job-metrics/td-p/42956" target="_blank">Monitoring job metrics - Databricks - 42956</A></P> Thu, 31 Aug 2023 10:20:50 GMT Bagger 2023-08-31T10:20:50Z https://community.databricks.com/t5/get-started-discussions/monitoring-job-metrics/m-p/42941#M853 <P>Hi,</P><P>We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.</P><P>Namely, we need to monitor the following:</P><UL><LI>failed jobs : is a job failed</LI><LI>table ingest rate : how much data is ingested</LI><LI>table ingest lag : is a stream job further behind than expected</LI><LI>table size : size of the current table being ingested into</LI><LI>query runtime : the time a query has been running</LI></UL><P>Does anyone have any ideas on how to get those metrics (either through Prometheus or an alternative method)?</P> Thu, 31 Aug 2023 08:22:11 GMT https://community.databricks.com/t5/get-started-discussions/monitoring-job-metrics/m-p/42941#M853 Bagger 2023-08-31T08:22:11Z https://community.databricks.com/t5/get-started-discussions/monitoring-job-metrics/m-p/42957#M854 <P>I have reposted this post in "Administation and Architecture"</P><P><A href="https://community.databricks.com/t5/administration-architecture/monitoring-job-metrics/td-p/42956" target="_blank">Monitoring job metrics - Databricks - 42956</A></P> Thu, 31 Aug 2023 10:20:50 GMT https://community.databricks.com/t5/get-started-discussions/monitoring-job-metrics/m-p/42957#M854 Bagger 2023-08-31T10:20:50Z