Hi @harvey-c, Certainly! Let’s explore how you can monitor and log row counts in your Delta Live Tables (DLT) pipelines.
Monitoring DLT Pipelines:
DLT provides built-in features for monitoring and observability. You can review most monitoring data manually through the pipeline details UI.
Here are some key aspects to consider:
- Pipeline Graph: The pipeline graph displays dependencies between datasets in your pipeline. By default, it shows the most recent update for the table. You can select older updates from a drop-down menu. Details displayed include pipeline ID, source libraries, compute cost, product edition, Databricks Runtime version, and the channel configured for the pipeline.
- List View: Click the “List” tab to see all datasets in your pipeline represented as rows in a table. This view is useful when your pipeline DAG is too large to visualize in the graph view.
- Dataset Details: Clicking on a dataset in the pipeline graph or dataset list displays schema information, data quality metrics, and a link back to the source code defining the dataset.
- Update History: To view the history and status of pipeline updates, use the update history drop-down menu. You can see the graph, details, and events for a specific update.
- Real-time Notifications: Set up email notifications for pipeline events, such as successful completion or failure of an update.
Row Count Validation:
You can add an additional table to your pipeline that defines an expectation to compare row counts between two live tables.
The results of this expectation appear in the event log and the DLT UI.
For example, to validate equal row counts between tables tbla and tblb, you can define an expectatio...:
- -- Validate row counts across tables SELECT COUNT(*) AS row_count_diff FROM tbla UNION ALL SELECT -COUNT(*) AS row_count_diff FROM tblb
Custom Logging:
If you need more customized logging, you can create your own logging statements within your DLT pipeline code.
For example, in Python, you can use the logging module to log messages. Here’s a snippet:
- import logging from datetime import datetime logger = logging.getLogger("raw_zone") logger.info("Processing of landing to Raw layer has started {0}".format(datetime.now()))
Adjust the logger name and log messages as needed for your specific use case.
Remember that the Delta Live Tables event log contains all information related to your pipeline, including audit logs, data quality checks, pipeline progress, and data lineage. You can use this event log to track, understand, and monitor the state of your data pipelines. Happy debugging! 🚀
