Is there a way to add a notification or set a metric threshold if a declarative pipeline (based on Auto Loader) ingests no data (I can see only max duration/backlog metrics in the workflow ui)?
Yes, there are several ways to detect and get notified when your Auto Loader pipeline ingests no data.
Here are the most effective approaches:
1. Streaming Backlog (Records) Metric
The Streaming backlog (records) metric you see in the UI can actually help detect no data scenarios:
- Set this metric to trigger when the backlog is consistently 0 for an extended period
- This indicates no new records are being processed
- Configure it with a threshold like "Alert if backlog = 0 for > 30 minutes"
2. Custom Metric Using Stream Metrics
Add a custom notification using Databricks' streaming metrics in your pipeline:
I'd suggest using option 2 (streaming metrics monitoring) combined with the Streaming backlog (records) metric threshold, as this gives you both real-time detection and built-in Databricks alerting capabilities.
The key is setting up proper thresholds based on your expected data ingestion patterns and business requirements.
