article Monitoring DLT Pipeline Performance and Best Practices in Technical Blog

article Monitoring DLT Pipeline Performance and Best Practices in Technical Blog https://community.databricks.com/t5/technical-blog/monitoring-dlt-pipeline-performance-and-best-practices/ba-p/118220 <H1><FONT size="6"><STRONG>Introduction - From Reactive Firefighting to Proactive Monitoring</STRONG></FONT></H1> <P class="lia-align-justify"><FONT size="3"><SPAN>In many industries, data pipelines power critical decisions from fraud detection to inventory management. When these pipelines fail silently or degrade gradually, this could cause downstream problems such as misplaced customer trust, compliance violations, and operational bottlenecks.</SPAN></FONT></P> <P class="lia-align-justify"><FONT size="3"><SPAN>Monitoring data workloads can be challenging if not done right, especially when issues are difficult to detect and resolve. Without adequate monitoring, teams may face unexpected downtime and spend significant time troubleshooting problems. DLT provides built-in monitoring features and integrates with Databricks’ native tools, making it easier to track pipeline health, identify bottlenecks, and respond quickly to errors. This approach helps ensure that data remains reliable and that pipelines run smoothly with minimal manual intervention.</SPAN></FONT></P> <P class="lia-align-justify"><FONT size="3"><SPAN>Unlike fragmented approaches that force teams to cobble together logs, DLT bakes monitoring into the following layers:</SPAN></FONT></P> <UL> <LI style="font-weight: 400;" aria-level="1"> <P class="lia-align-justify"><FONT size="3"><STRONG>Centralized data quality rules</STRONG><SPAN>: This lets the team embed checks like “<EM>transaction amounts must be positive</EM>” directly into pipeline code, ensuring validation happens at the source.</SPAN></FONT></P> </LI> <LI style="font-weight: 400;" aria-level="1"> <P class="lia-align-justify"><FONT size="3"><STRONG>Streaming metrics</STRONG><SPAN>: DLT pipelines can now monitor streaming sources such as Apache Kafka and Auto Loader. Streaming metrics include backlog bytes, records, seconds, and files, which are displayed in real-time within the DLT UI.</SPAN></FONT></P> </LI> <LI style="font-weight: 400;" aria-level="1"> <P class="lia-align-justify"><FONT size="3"><STRONG>Automatic Lineage mapping</STRONG><SPAN>: This visually traces errors back to their origin.</SPAN></FONT></P> </LI> <LI style="font-weight: 400;" aria-level="1"> <P class="lia-align-justify"><FONT size="3"><STRONG>Custom Monitoring</STRONG><SPAN>: New capabilities allow for defining custom actions based on specific pipeline events. This is particularly useful for alerts on operational thresholds and other monitoring flexibilities. In addition to these native capabilities, teams can still leverage traditional Spark mechanisms such as </SPAN><FONT face="courier new,courier" color="#339966">StreamingQueryListener</FONT><SPAN>, cluster logs, and driver/executor logs for deeper insights, troubleshooting, and historical auditing.</SPAN></FONT></P> </LI> <LI style="font-weight: 400;" aria-level="1"> <P class="lia-align-justify"><FONT size="3"><STRONG>Self-correcting workflows</STRONG><SPAN>: DLT can automatically retry failed tasks or trigger alerts for unresolved issues, minimizing manual firefighting. If problems arise, DLT automatically maps errors back to their origin-for example, tracing a sudden spike in invalid customer IDs to a specific transformation step. For common hiccups like brief network outages, the system quietly retries tasks up to five times, alerting engineers only if problems persist.</SPAN></FONT></P> </LI> </UL> <P class="lia-align-justify"><FONT size="3"><SPAN>In this article, we explore performance-focused monitoring best practices, highlighted through a real-life deployment example.</SPAN></FONT></P> <P class="lia-align-justify"> </P> <H1><FONT size="6">1. Monitoring Techniques with DLT Pipelines</FONT></H1> <P class="lia-align-justify"><SPAN>As mentioned above, DLT has built-in monitoring, which includes DLT UI quality metrics, structured event logs, and system-level metadata tables. This allows data engineers to track pipeline health in a scalable way.</SPAN></P> <H2><SPAN>1.1 DLT Monitoring Features</SPAN></H2> <P class="lia-align-justify"><FONT size="3"><SPAN>Consider the following DLT Python logic that moves sample quote data from Bronze to Gold. We will use this example to guide our discussion and explore how the DLT UI and DLT expectations can help monitor the quality of the data being processed.</SPAN></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Quotes Processing Medallion Architecture" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16651iCC060A12EBF3AE74/image-size/large?v=v2&px=999" role="button" title="image.png" alt="Quotes Processing Medallion Architecture" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Quotes Processing Medallion Architecture</span></span></P> <P><SPAN>Below is the code block to read the data from the source table, </SPAN><SPAN><FONT face="courier new,courier" color="#339966">main.pricing_data.quotes</FONT>:</SPAN></P> <LI-CODE lang="python">import dlt from pyspark.sql.functions import * @dlt.table( comment="Raw quotes data", table_properties={"quality": "bronze"}) def bronze_quotes(): return spark.sql("select * from main.pricing_data.quotes") </LI-CODE> <P><FONT size="3"><SPAN>The next block of code is used to transform the bronze data into silver and gold layers:</SPAN></FONT></P> <UL> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><FONT size="3"><SPAN>The silver hop is to add expectations on the bronze data, which tells DLT to expect a certain quality of data to pass through downstream. If there is any violation, we can control what happens to the insufficient data.</SPAN></FONT></LI> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><FONT size="3"><SPAN>Finally, in the gold layer, we perform aggregations and make the data publish-ready.</SPAN></FONT></LI> </UL> <LI-CODE lang="python"># Silver Layer Table @dlt.table(table_properties={"quality": "silver"}) @dlt.expect_or_drop("valid_pricing", "Quote_Price < 3900") @dlt.expect_or_drop("valid_driver_age", "Driver_Age BETWEEN 17 AND 90") def silver_quotes(): return ( dlt.read("bronze_quotes") .dropDuplicates(["ID"]) .filter( (col("Annual_Mileage") > 0) & (col("Credit_Score").between(300, 850)) & (col("Vehicle_Type").isin(["Car", "Truck", "Motorbike"]))) .withColumn("Quote_Price", col("Quote_Price").cast("integer")) .withColumn("Driver_Age", col("Driver_Age").cast("integer")) .withColumn("Date", col("Date").cast("date"))) # Gold Layer Table @dlt.table(table_properties={"quality": "gold"}) def gold_vehicle_quotes(): return ( dlt.read("silver_quotes") .groupBy("Vehicle_Type") .agg( avg("Quote_Price").alias("Average_Premium"), count("ID").alias("Total_Policies"))) </LI-CODE> <H3>1.1.1 UI Quality Metrics</H3> <P class="lia-align-justify"><FONT size="3"><SPAN>The default feature of observability in the DLT pipeline is achieved using a high-quality lineage diagram that provides visibility into how data flows for impact analysis.</SPAN></FONT></P> <P class="lia-align-justify"><FONT size="3"><SPAN>When the pipeline is executed, the DLT UI displays the corresponding DAG, which serves as the first level of monitoring. This view provides an overview of the data volume being processed, particularly in the Silver layer. In this example, approximately 42k records were processed successfully, while around 8.3k records failed the quality checks.</SPAN></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="DLT Pipeline DAG - Quotes Processing" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16650iD66453391B508F97/image-size/large?v=v2&px=999" role="button" title="image.png" alt="DLT Pipeline DAG - Quotes Processing" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">DLT Pipeline DAG - Quotes Processing</span></span></P> <P class="lia-align-justify"><SPAN>If we click the </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>silver_quotes</SPAN></FONT><SPAN> materialized view, we can check out the “</SPAN><STRONG>Data Quality</STRONG><SPAN>” tab, which shows more granular information. It shows the user the number of records written into the target and those dropped. Additionally, it also indicates the expectations, namely </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>valid_driver_age</SPAN></FONT><SPAN> and </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>valid_pricing</SPAN></FONT><SPAN>,</SPAN><SPAN> which drove these metrics.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Data Quality Statistics on silver_quotes table" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16649iC4121839F1A4D455/image-size/medium?v=v2&px=400" role="button" title="image.png" alt="Data Quality Statistics on silver_quotes table" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Data Quality Statistics on silver_quotes table</span></span><SPAN> </SPAN></P> <H3>1.1.2 DLT Expectations</H3> <P class="lia-align-justify"><SPAN>DLT enforces data quality through declarative expectations that validate records during pipeline execution. The DLT UI's </SPAN><STRONG>Data Quality</STRONG><SPAN> tab displays real-time metrics for constraints defined via </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>expect</SPAN></FONT><SPAN> (retain), </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>expect_or_drop</SPAN></FONT><SPAN>, and </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>expect_or_fail</SPAN></FONT><SPAN> operators. For example, we have used the following operators to drop records that do not meet the valid cut-off age for people obtaining a driver’s quotation and to filter out high quote price values using the </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>@expect_or_drop</SPAN></FONT><SPAN> decorator on the Silver materialized view, </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>silver_quotes</SPAN></FONT><SPAN>.</SPAN></P> <LI-CODE lang="python"># valid pricing expectation checks for the value of quote price @dlt.expect_or_drop("valid_pricing", "Quote_Price < 3900") # valid driver age expectation checks for the age of the driver @dlt.expect_or_drop("valid_driver_age", "Driver_Age BETWEEN 17 AND 90")</LI-CODE> <P class="lia-align-justify"><SPAN>More information and sample DLT exceptions can be found in this Databricks official documentation: </SPAN><A href="https://docs.databricks.com/aws/en/dlt/expectations" target="_blank" rel="noopener">Manage data quality with pipeline expectations</A></P> <H2><SPAN>1.2 Event Log</SPAN></H2> <P class="lia-align-justify"><SPAN>Each DLT pipeline automatically creates an event log related to a pipeline, including audit logs, data quality checks, pipeline progress, and data lineage.</SPAN></P> <P class="lia-align-justify"><SPAN>There are two ways to access this DLT </SPAN><A href="https://docs.databricks.com/aws/en/dlt/observability#what-is-the-dlt-event-log" target="_blank" rel="noopener"><SPAN>event log</SPAN></A><SPAN>: through the <STRONG>default publishing mode</STRONG> or the <STRONG>legacy publishing mode</STRONG>.</SPAN></P> <P class="lia-align-justify" data-unlink="true"><SPAN>With default publishing mode, each DLT pipeline automatically creates a log capturing lifecycle events, expectation outcomes, errors, and operational metadata. If you’re building your pipeline via the Databricks UI, you can set this under “<STRONG>Advanced settings” </STRONG>in the pipeline configuration.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="ETL Pipeline Create Configuration" style="width: 781px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16643i9EB6B929C8A9B8C9/image-size/large?v=v2&px=999" role="button" title="image.png" alt="ETL Pipeline Create Configuration" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">ETL Pipeline Create Configuration</span></span></P> <P class="lia-align-justify"><SPAN>If using JSON or API-based configuration, you can add the following code within your pipeline JSON script using:</SPAN></P> <LI-CODE lang="python">"event_log": { "catalog": "main", "schema": "monitoring", "name": "event_log_quotes_pipeline" }</LI-CODE> <P class="lia-align-justify"><SPAN>This configuration will publish structured pipeline events into the specified Unity Catalog location under the catalog and schema specified, making them queryable like any other Delta table. </SPAN><SPAN>The example below filters for error-level events from the past 24 hours, helping you quickly identify recent pipeline failures, expectation violations, or operational issues.</SPAN></P> <LI-CODE lang="python">SELECT timestamp, event_type, message FROM main.monitoring.event_log_quotes_pipeline WHERE level = 'ERROR' AND timestamp > current_timestamp() - INTERVAL 24 HOURS ORDER BY timestamp DESC;</LI-CODE> <P class="lia-align-justify"><SPAN>With legacy publishing mode, you can utilize the </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>event_log</SPAN></FONT><SPAN> Table-Valued function (TVF) to fetch the event log for the pipeline. You retrieve the event log for a pipeline by passing the pipeline ID or a table name to the TVF. For example, to recover the event log records for the pipeline with ID "</SPAN><FONT face="courier new,courier" color="#339966"><SPAN>19c78331-2ea5-1973-a1b2-4dbef8b2198c</SPAN></FONT><SPAN>", you can run the following:</SPAN></P> <LI-CODE lang="python">SELECT * FROM event_log("event_log_19c78331_2ea5_1973_a1b2_4dbef8b2198c")</LI-CODE> <P class="lia-align-justify"><SPAN>You can see if your pipeline is in legacy publishing mode if you see the following in your pipeline UI settings:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="LingeshK_1-1746630980490.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16629i4F94AE86FD2202CB/image-size/medium?v=v2&px=400" role="button" title="LingeshK_1-1746630980490.png" alt="LingeshK_1-1746630980490.png" /></span></P> <P class="lia-align-justify"><SPAN>Additionally, for legacy publishing mode, remember the following pre-requisites:</SPAN></P> <UL> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>event_log</SPAN></FONT><SPAN> TVF can be called only by the pipeline owner.</SPAN></LI> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><SPAN>You cannot use the </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>event_log</SPAN></FONT><SPAN> table-valued function in a pipeline or query to access the event logs of multiple pipelines.</SPAN></LI> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><SPAN>You cannot share a view created over the </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>event_log</SPAN></FONT><SPAN> table-valued function with other users.</SPAN></LI> </UL> <P class="lia-align-justify"><SPAN>This diagram illustrates how DLT in Databricks offers unified observability by combining visual pipeline monitoring, configurable event logging, and structured audit trails. A sample DLT pipeline with bronze, Silver, and gold layers shows real-time execution status and data volumes. Event log configuration allows users to define where logs are stored, such as in the Unity Catalog under the </SPAN><FONT color="#339966"><SPAN>main.monitoring.event_log_sales_pipeline</SPAN></FONT><SPAN>. The DLT UI displays a livestream of pipeline events, while the underlying Delta table captures structured log data including event type, timestamp, and user actions. This integration enables both immediate visual feedback and advanced programmatic monitoring, streamlining debugging, auditing, and alerting workflows.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="DLT Monitoring Event Configuration" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16652i92278B49FD955CDF/image-size/large?v=v2&px=999" role="button" title="image.png" alt="DLT Monitoring Event Configuration" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">DLT Monitoring Event Configuration</span></span></P> <H3>1.2.1 Dissecting the Event Logs for Non-Technical Personas</H3> <P class="lia-align-justify"><SPAN>We can navigate to Genie space using the “</SPAN><STRONG>Ask Genie</STRONG><SPAN>” option in the Sample data pane enables users to interact with their datasets using natural language queries. We can ask queries as shown in the snippet below:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Sales Event Log Table Analysis via Genie Space" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16653i20359967C589FF8E/image-size/large?v=v2&px=999" role="button" title="image.png" alt="Sales Event Log Table Analysis via Genie Space" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Sales Event Log Table Analysis via Genie Space</span></span><SPAN> </SPAN></P> <P class="lia-align-justify"><SPAN>Genie dynamically adjusts its responses according to user feedback, which can be verified via the ‘</SPAN><STRONG>Show code</STRONG><SPAN>’ option.</SPAN></P> <H3><SPAN>1.2.2 Dissecting the Event Logs using the SQL Editor</SPAN></H3> <P class="lia-align-justify"><SPAN>Users can sometimes use the SQL Editor to create more granular queries that could power the AI/BI dashboards.</SPAN></P> <P class="lia-align-justify"><SPAN>Consider this scenario - the user wants to get the count of the passing and failing records for all the Data Quality checks that are set in the DLT code. Then, the following query is one way of achieving this:</SPAN></P> <LI-CODE lang="python">-- Create or replace a temporary view named latest_update that selects the most recent 'create_update' event CREATE OR REPLACE TEMP VIEW latest_update AS SELECT origin.update_id AS id FROM `event_log_sales_pipeline` WHERE event_type = 'create_update' ORDER BY timestamp DESC LIMIT 1; -- Select dataset, expectation name, and the sum of passing and failing records for each expectation SELECT row_expectations.dataset as dataset, row_expectations.name as expectation, SUM(row_expectations.passed_records) as passing_records, SUM(row_expectations.failed_records) as failing_records FROM ( -- Explode the JSON array of expectations into individual rows SELECT explode( from_json( details :flow_progress :data_quality :expectations, "array<struct<name: string, dataset: string, passed_records: int, failed_records: int>>" ) ) row_expectations FROM `event_log_sales_pipeline`, latest_update WHERE event_type = 'flow_progress' AND origin.update_id = latest_update.id ) GROUP BY row_expectations.dataset, row_expectations.name;</LI-CODE> <P class="lia-align-justify"><SPAN>This SQL query leverages the custom event log table (</SPAN><FONT face="courier new,courier" color="#339966"><SPAN>event_log_sales_pipeline</SPAN></FONT><SPAN>) in the metastore to derive data quality metrics by analyzing the latest pipeline updates. Specifically:</SPAN></P> <UL> <LI class="lia-align-left" style="font-weight: 400;" aria-level="1"><SPAN>It extracts and aggregates metrics related to expectations, such as </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>passing_records</SPAN></FONT><SPAN> and </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>failing_records</SPAN></FONT><SPAN> counts, from the following JSON field:</SPAN></LI> </UL> <PRE><STRONG><FONT face="courier new,courier" color="#339966">details:flow_progress:data_quality:expectations</FONT></STRONG></PRE> <UL> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><SPAN>It ensures analysis is focused on the most recent pipeline update by filtering using </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>event_type = 'flow_progress'</SPAN></FONT><SPAN> and matching </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>update_id</SPAN></FONT><SPAN> with the latest value in the </SPAN><FONT face="courier new,courier" color="#339966"><SPAN>latest_update</SPAN></FONT><SPAN> temporary view.</SPAN></LI> <LI class="lia-align-justify" style="font-weight: 400;" aria-level="1"><SPAN>Results are grouped by the dataset and expectation name, providing a concise summary of data quality performance per dataset.</SPAN></LI> </UL> <P class="lia-align-justify"><SPAN>This is the result generated from this query:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Pass & Fail Record Count Split on silver_quotes table" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16654iD3B18A25F6B208C9/image-size/large?v=v2&px=999" role="button" title="image.png" alt="Pass & Fail Record Count Split on silver_quotes table" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Pass & Fail Record Count Split on silver_quotes table</span></span></P> <P class="lia-align-justify"><SPAN>In another scenario, if your DLT pipeline stores the output in HMS, you can query the event log delta table by pointing to the event log folder (event logs are captured and stored in the system/event folder under the storage location of the DLT pipeline).</SPAN></P> <LI-CODE lang="python">SELECT * FROM delta.`<event_log_path>`</LI-CODE> <P class="lia-align-justify"><SPAN>If the user wants to query the top 10 most frequent exception class names, they can run the following command:</SPAN></P> <LI-CODE lang="python">-- Select the top 10 most frequent exception class names from the exploded error.exceptions array SELECT exception.class_name, COUNT(*) AS occurrence FROM main.monitoring.event_log_sales_pipeline LATERAL VIEW EXPLODE(error.exceptions) AS exception GROUP BY exception.class_name ORDER BY occurrence DESC LIMIT 10; -- you can change this value to get more results</LI-CODE> <P><SPAN>This is a sample result from the query above:</SPAN><SPAN><BR /></SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Frequency of Error Classes generated from Event Log Table" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16655iB3C44E78CFA7A20E/image-size/large?v=v2&px=999" role="button" title="image.png" alt="Frequency of Error Classes generated from Event Log Table" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Frequency of Error Classes generated from Event Log Table</span></span></P> <P class="lia-align-justify"><SPAN>To measure the average interval between events for a specific pipeline to monitor activity frequency or detect lulls, we can write the following query:</SPAN></P> <LI-CODE lang="python">WITH event_times AS ( -- Selects timestamps from the event log for the 'Quotes Processing Pipeline' and orders them chronologically SELECT timestamp FROM main.monitoring.event_log_sales_pipeline WHERE origin.pipeline_name = 'Quotes Processing Pipeline' ORDER BY timestamp ), diffs AS ( -- Calculates the difference between consecutive timestamps using the LEAD function SELECT timestamp, LEAD(timestamp) OVER (ORDER BY timestamp) AS next_timestamp FROM event_times ) -- Computes the average time difference between consecutive events in seconds SELECT CONCAT(ROUND(AVG(UNIX_TIMESTAMP(next_timestamp) - UNIX_TIMESTAMP(timestamp))/1000, 2), ' seconds') AS avg_seconds_between_events FROM diffs WHERE next_timestamp IS NOT NULL;</LI-CODE> <P class="lia-align-justify"><SPAN>This will yield a result of the average time interval between all the events that are getting logged:</SPAN></P> <P class="lia-align-justify"><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Average Latency (seconds) between two consecutive events" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16656iDB0A5984C3360408/image-size/medium?v=v2&px=400" role="button" title="image.png" alt="Average Latency (seconds) between two consecutive events" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Average Latency (seconds) between two consecutive events</span></span></SPAN><SPAN>Finally, we can make use of the <FONT face="courier new,courier" color="#339966">event_log</FONT> table’s </SPAN><SPAN>maturity</SPAN><SPAN> property to indicate the stability of the event schema:</SPAN></P> <LI-CODE lang="python">-- This query selects the maturity level of events, counts the total number of events for each maturity level, -- and sums the number of fatal errors for each maturity level. The results are grouped by maturity level and -- ordered by the total number of events in descending order. SELECT maturity_level, COUNT(*) AS total_events, SUM(CASE WHEN error.fatal THEN 1 ELSE 0 END) AS fatal_errors FROM main.monitoring.event_log_sales_pipeline GROUP BY maturity_level ORDER BY total_events DESC;</LI-CODE> <P class="lia-align-justify"><SPAN>This will yield the following result:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="LingeshK_1-1746632421702.png" style="width: 730px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16637i6154E9ED4DBA47DE/image-dimensions/730x123?v=v2" width="730" height="123" role="button" title="LingeshK_1-1746632421702.png" alt="LingeshK_1-1746632421702.png" /></span></P> <P class="lia-align-justify"><SPAN>Maturity levels, introduced in </SPAN><A href="https://docs.databricks.com/gcp/en/release-notes/dlt/2022/37/#:~:text=Event%20log%20entries%20now%20contain%20the%20maturity%20property%20to%20indicate%20the%20stability%20of%20the%20event%20schema" target="_blank" rel="noopener"><SPAN>DLT release 2022.37</SPAN></A><SPAN>, indicate the stability of event schemas within the pipeline's event log. </SPAN><FONT face="courier new,courier" color="#008000"><SPAN>STABLE</SPAN></FONT><SPAN> events have guaranteed schemas that will not change across DLT versions, making them dependable anchors for monitoring solutions. In contrast, </SPAN><FONT face="courier new,courier" color="#008000"><SPAN>EVOLVING</SPAN></FONT><SPAN> events have schemas that may change in future releases.</SPAN></P> <P class="lia-align-justify"><SPAN>The higher proportion of </SPAN><FONT face="courier new,courier" color="#008000"><SPAN>STABLE</SPAN></FONT><SPAN> events (65) versus </SPAN><FONT face="courier new,courier" color="#008000"><SPAN>EVOLVING</SPAN></FONT><SPAN> events (17) suggests this pipeline has settled chiefly into reliable event patterns, which is preferable for production environments.</SPAN></P> <P class="lia-align-justify"><SPAN>Utilizing the built-in </SPAN><FONT face="courier new,courier" color="#008000"><SPAN>event_log</SPAN></FONT><SPAN> table in DLT is helpful for monitoring and identifying data quality trends in the pipelines, offering insights into the effectiveness of expectations defined within the system.</SPAN></P> <H3><SPAN>1.2.3 Building Dashboards from the DLT Event Log Table</SPAN></H3> <P class="lia-align-justify"><SPAN>You can then embed these queries into the AI/BI dashboard as custom SQL queries in the “<STRONG>Data</STRONG>” pane. One simple example is to set the dataset to the custom event log table and use either the assistant or the visualization widget toolbar to create appropriate visualizations.</SPAN></P> <LI-CODE lang="python">SELECT * FROM main.monitoring.event_log_sales_pipeline;</LI-CODE> <P class="lia-align-justify"><SPAN>Below is an example of how the visualizations are built on top of the dataset that captures all the records of the </SPAN><FONT face="courier new,courier" color="#008000"><SPAN>main.monitoring.event_log_sales_pipeline</SPAN></FONT><SPAN> event log table.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Event Log Sales Pipeline Dashboard" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16657iADE070EE84D76B84/image-size/large?v=v2&px=999" role="button" title="image.png" alt="Event Log Sales Pipeline Dashboard" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Event Log Sales Pipeline Dashboard</span></span></P> <H2>1.3 System Tables for Monitoring</H2> <P class="lia-align-justify"><SPAN>Databricks </SPAN><A href="https://docs.databricks.com/aws/en/admin/system-tables#which-system-tables-are-available" target="_blank" rel="noopener"><SPAN>system tables</SPAN></A><SPAN> offer structured metadata across pipelines, jobs, and clusters, enabling easy alerting using SQL. These tables can be queried directly or integrated with dashboards, alerts, and workflows to automate monitoring. </SPAN><SPAN>Below are some common system tables used for monitoring DLT pipelines: </SPAN></P> <UL> <LI aria-level="1"><A href="https://docs.databricks.com/aws/en/admin/system-tables/audit-logs" target="_self"><FONT face="courier new,courier" color="#008000"><STRONG>system.access.audit</STRONG></FONT></A></LI> <LI aria-level="1"><A href="https://docs.databricks.com/aws/en/admin/system-tables/billing" target="_self"><FONT face="courier new,courier" color="#008000"><STRONG>system.billing.usage</STRONG></FONT></A></LI> <LI aria-level="1"><A href="https://docs.databricks.com/aws/en/admin/system-tables/pricing" target="_self"><FONT face="courier new,courier" color="#008000"><STRONG>system.billing.list_prices</STRONG></FONT></A></LI> </UL> <P><LI-WRAPPER></LI-WRAPPER></P> <P class="lia-align-justify"><SPAN>These tables can be queried directly or integrated with dashboards, alerts, and workflows to automate monitoring and incident response.</SPAN></P> <P class="lia-align-justify"> </P> <H1><FONT size="6"><STRONG>2. </STRONG><STRONG>From Metrics to Monitoring: Internal Observability Tools</STRONG></FONT></H1> <P class="lia-align-justify">Once your event logs are available in Unity Catalog (see previous section for setup and queries), you can query and display them with visual dashboards, alerts, streaming tables, and third-party tools.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="LingeshK_0-1746632785660.png" style="width: 698px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16640i8B8812B1FCC053D4/image-dimensions/698x301?v=v2" width="698" height="301" role="button" title="LingeshK_0-1746632785660.png" alt="LingeshK_0-1746632785660.png" /></span></P> <H2><STRONG>2.1 Monitoring Metrics within Databricks</STRONG></H2> <P class="lia-align-justify"><SPAN>Below are three practical ways to bring your metrics into actionable monitoring.</SPAN></P> <H3><STRONG>2.1.1 Visualise with AI/BI Databricks Dashboards</STRONG></H3> <P class="lia-align-justify"><SPAN>You can turn your SQL queries (e.g., pipeline error counts or failure statuses) into </SPAN><A href="https://www.databricks.com/product/business-intelligence" target="_blank" rel="noopener"><SPAN>AI/BI dashboards</SPAN></A><SPAN>. Users can build charts and graphs to analyze the pipelines. Check out our dedicated </SPAN><A href="https://www.databricks.com/resources/demos/tutorials#:~:text=retail%2Dc360%27)-,Explore%20all%20tutorials,-Lakehouse%20Platform" target="_blank" rel="noopener"><SPAN>DBdemos</SPAN></A><SPAN> page to forge a similar dashboard as shown below.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="LingeshK_1-1746632836295.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16641i327AFFF7A2F938D7/image-size/large?v=v2&px=999" role="button" title="LingeshK_1-1746632836295.png" alt="LingeshK_1-1746632836295.png" /></span></P> <H3> </H3> <H3><STRONG>2.1.2 Set up SQL Alerts for Proactive Monitoring</STRONG></H3> <P><SPAN>Another helpful Databricks feature is </SPAN><A href="https://docs.databricks.com/aws/en/sql/user/alerts/" target="_blank" rel="noopener"><SPAN>SQL alerts</SPAN></A><SPAN>. You can set up these alerts to monitor any pipeline discrepancies (i.e., table count exceeding the expected limit)  and send notifications via SQL alerts or </SPAN><A href="https://docs.databricks.com/aws/en/admin/workspace-settings/notification-destinations" target="_blank" rel="noopener"><SPAN>webhooks</SPAN></A><SPAN>.</SPAN></P> <H3> </H3> <H3><STRONG>2.1.3 Build Streaming Tables for Monitoring</STRONG></H3> <P><SPAN>For continuous monitoring, you can convert the event log into a streaming table. This is especially useful when you want to detect and respond to issues, such as failed jobs and error messages, as soon as they appear in the logs.  </SPAN></P> <P><SPAN>Below is an example script on how to create an event log streaming table via a notebook:</SPAN></P> <LI-CODE lang="python">df = spark.readStream.table("main.monitoring.event_log_sales_pipeline") \ .filter("level = 'ERROR'") df.writeStream \ .format("delta") \ .option("checkpointLocation","/Volumes/main/monitoring/event_log_errors_stream_checkpoint") \ .outputMode("append") \ .table("main.monitoring.event_log_errors_stream")</LI-CODE> <P class="lia-align-justify"><SPAN>Instead of directly querying the raw event log table repeatedly, which can be inefficient, you can create a dedicated streaming table with a filtered subset (e.g., error-level events). This has several advantages:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Improved performance</STRONG><SPAN>: Queries on smaller, filtered tables are faster and more efficient.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Data isolation</STRONG><SPAN>: Only the relevant data (e.g., errors) is shared, keeping the whole event log private.</SPAN></LI> </UL> <P class="lia-align-justify"><SPAN>Note: Although this is a streaming table, updates will only occur after the pipeline has finished running. This is because the DLT event log is written at the end of each pipeline run, not continuously during execution.</SPAN></P> <P class="lia-align-justify"> </P> <H1><FONT size="6"><STRONG>3. </STRONG><STRONG>Example Scenario</STRONG></FONT></H1> <P class="lia-align-justify"><SPAN>A major retail chain implemented DLT to process its daily sales data across thousands of stores nationwide. Their pipeline followed the medallion architecture shown in the diagram: ingesting raw sales data from cloud storage and message queues into Bronze tables, transforming it into validated customer and order records in Silver, and finally creating aggregated daily sales views in the Gold layer for business consumption.</SPAN></P> <P class="lia-align-justify"><SPAN>Initially, the pipeline operated smoothly, but as data volumes grew with the business expansion, the operations team began noticing concerning patterns. Daily pipeline runs were increasingly failing during peak sales periods, particularly following weekend promotions when transaction volumes spiked significantly.</SPAN></P> <P class="lia-align-justify"><SPAN>The team investigated using DLT’s built-in monitoring capabilities. Analyzing the DLT event log uncovered a correlation between cost spikes and excessive autoscaling behaviour in their cluster configuration.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="High Level End-to-End flow of DLT pipeline Monitoring Process" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16658i4FEC193FA880D7C9/image-size/large?v=v2&px=999" role="button" title="image.png" alt="High Level End-to-End flow of DLT pipeline Monitoring Process" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">High Level End-to-End flow of DLT pipeline Monitoring Process</span></span></P> <H1> </H1> <H1><FONT size="6"><STRONG>4. </STRONG><STRONG>Conclusion</STRONG></FONT></H1> <P class="lia-align-justify"><SPAN>Don’t let monitoring be an afterthought. Leverage our observability capabilities to monitor pipeline performance and data quality anomalies. </SPAN></P> <P class="lia-align-justify"><SPAN><span class="lia-unicode-emoji" title=":rocket:">🚀</span></SPAN><STRONG> Take action:</STRONG><SPAN> Start implementing pipeline monitoring, leverage the event log, and integrate with your observability stack to unlock the full power of DLT today!</SPAN></P> <P class="lia-align-justify"><SPAN>Check out the <A href="https://community.databricks.com/" target="_blank" rel="noopener">Databricks Community</A> for even more exciting blogs about the latest features of Databricks.</SPAN></P> <P class="lia-align-justify"> </P> <H1><FONT size="6"><STRONG>5. </STRONG><STRONG>Resources</STRONG></FONT></H1> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/gcp/en/dlt/observability" target="_blank" rel="noopener"><SPAN>Monitor DLT pipelines | Databricks Documentation</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.azure.cn/en-us/databricks/delta-live-tables/event-hooks" target="_blank" rel="noopener"><SPAN>Implement custom monitoring of DLT pipelines with event hooks (Public Preview)</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://www.databricks.com/blog/introducing-streaming-observability-workflows-and-dlt-pipelines" target="_blank" rel="noopener"><SPAN>Introducing Streaming Observability in Workflows and DLT Pipelines</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/dlt/" target="_blank" rel="noopener"><SPAN>What is DLT? | Databricks Documentation</SPAN></A></LI> </UL> Mon, 19 May 2025 08:23:46 GMT LingeshK 2025-05-19T08:23:46Z Monitoring DLT Pipeline Performance and Best Practices https://community.databricks.com/t5/technical-blog/monitoring-dlt-pipeline-performance-and-best-practices/ba-p/118220 <P><SPAN>Stop reactive firefighting! Our comprehensive guide <span class="lia-unicode-emoji" title=":ledger:">📒</span> will teach you how to proactively monitor Databricks DLT pipelines for optimal performance and data quality</SPAN></P> Mon, 19 May 2025 08:23:46 GMT https://community.databricks.com/t5/technical-blog/monitoring-dlt-pipeline-performance-and-best-practices/ba-p/118220 LingeshK 2025-05-19T08:23:46Z