cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Monitoring

Revathy123
New Contributor II

Hi Everyone,

can someone suggest me to select best native job monitoring tool available in Databricks for fulfill my need;

we need to monitor the following:

Number of failed jobs and its name : for last 24 hours
Table that are not getting data
Latest ingested timestamp
table ingest rate, how much data is ingested
table ingest lag,  is a stream job further behind than expected
table size, size of the current table being ingested into
query runtime : the time a query has been running
Thanks in advance

2 ACCEPTED SOLUTIONS

Accepted Solutions

Kaniz
Community Manager
Community Manager

Hi @Revathy123 , Databricks provides robust functionality for monitoring custom application metrics, streaming query events, and application log messages.


- Databricks offers native job monitoring tools.
- Databricks Monitoring allows you to track various metrics and events related to your jobs.
- You can monitor the number of failed jobs and their names.
- You can monitor tables that are not receiving data.
- Databricks provides metrics to track the latest ingested timestamp.
- You can monitor the rate at which data is being ingested into tables.
- Databricks allows you to monitor if a stream job is falling behind the expected ingest rate.
- You can monitor the size of the current table being ingested into.
- Databricks provides metrics to track the runtime of queries.
- You can send monitoring data from Databricks to Azure Monitor.
- Azure Monitor provides advanced monitoring and alerting capabilities for Databricks applications.
- If you are using Databricks on AWS, you can integrate it with CloudWatch for monitoring.
- CloudWatch enables you to derive metrics from logs and set up alerts.

View solution in original post

Revathy123
New Contributor II

@KanizI an new to this could you please help me to understand how we can achieve all those . Is databricks job API will help me to achieve this?

View solution in original post

4 REPLIES 4

Kaniz
Community Manager
Community Manager

Hi @Revathy123 , Databricks provides robust functionality for monitoring custom application metrics, streaming query events, and application log messages.


- Databricks offers native job monitoring tools.
- Databricks Monitoring allows you to track various metrics and events related to your jobs.
- You can monitor the number of failed jobs and their names.
- You can monitor tables that are not receiving data.
- Databricks provides metrics to track the latest ingested timestamp.
- You can monitor the rate at which data is being ingested into tables.
- Databricks allows you to monitor if a stream job is falling behind the expected ingest rate.
- You can monitor the size of the current table being ingested into.
- Databricks provides metrics to track the runtime of queries.
- You can send monitoring data from Databricks to Azure Monitor.
- Azure Monitor provides advanced monitoring and alerting capabilities for Databricks applications.
- If you are using Databricks on AWS, you can integrate it with CloudWatch for monitoring.
- CloudWatch enables you to derive metrics from logs and set up alerts.

Revathy123
New Contributor II

@KanizI an new to this could you please help me to understand how we can achieve all those . Is databricks job API will help me to achieve this?

Can someone help suggest me which monitoring tool helps me and how we can  achieve.

BilalAslamDbrx
Honored Contributor II
Honored Contributor II

  let me go through these one by one: 

Number of failed jobs and its name : for last 24 hours

[BA] Your best bet will be to use the upcoming system tables integration. This is in preview, I believe. The general idea is that you will get a Delta table with runs and their statuses. For now, you can also use the Job Runs page (Workflows > Job Runs), this will show you job runs and their failures (also accessible by API)

Table that are not getting data
[BA] Lakehouse Monitoring is the way to go! Also in Preview, I believe.

Latest ingested timestamp
table ingest rate, how much data is ingested

[BA] How are you ingesting data?

table ingest lag,  is a stream job further behind than expected

[BA] We are working on this! You will get monitoring and alerting on streaming lag both in Structured Streaming and Delta Live Tables.

table size, size of the current table being ingested into

[BA] I'm not sure what you mean by this.

query runtime : the time a query has been running

[BA] What type of query are you interested in?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.