Notifications for Data Ingestion in Declarative Pipelines Using Auto Loader

lucami — Wed, 02 Jul 2025 13:53:56 GMT

Is there a way to add a notification or set a metric threshold if a declarative pipeline (based on Auto Loader) ingests no data (I can see only max duration/backlog metrics in the workflow ui)?

Re: Notifications for Data Ingestion in Declarative Pipelines Using Auto Loader

lingareddy_Alva — Wed, 02 Jul 2025 19:16:50 GMT

Yes, there are several ways to detect and get notified when your Auto Loader pipeline ingests no data.
Here are the most effective approaches:

1. Streaming Backlog (Records) Metric
The Streaming backlog (records) metric you see in the UI can actually help detect no data scenarios:
- Set this metric to trigger when the backlog is consistently 0 for an extended period
- This indicates no new records are being processed
- Configure it with a threshold like "Alert if backlog = 0 for > 30 minutes"
2. Custom Metric Using Stream Metrics
Add a custom notification using Databricks' streaming metrics in your pipeline:

I'd suggest using option 2 (streaming metrics monitoring) combined with the Streaming backlog (records) metric threshold, as this gives you both real-time detection and built-in Databricks alerting capabilities.
The key is setting up proper thresholds based on your expected data ingestion patterns and business requirements.

topic Notifications for Data Ingestion in Declarative Pipelines Using Auto Loader in Data Engineering

Notifications for Data Ingestion in Declarative Pipelines Using Auto Loader

Re: Notifications for Data Ingestion in Declarative Pipelines Using Auto Loader