- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-30-2024 06:27 AM
Hi @jeremy98,
To read the Change Data Feed (CDF) logs in materialized views created by a Delta Live Tables (DLT) pipeline, you can follow these steps:
Enable Change Data Feed: Ensure that the change data feed is enabled on the base tables of the materialized views. You can enable it using the following SQL command:
ALTER TABLE table1 SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
View the Refresh History: You can view the status of refresh operations on a materialized view, including current and past refreshes, by querying the Delta Live Tables event log. Use the following SQL query to retrieve the relevant information:
SELECT *
FROM event_log(TABLE(<fully-qualified-table-name>))
WHERE event_type = 'update_progress'
ORDER BY timestamp DESC;
Replace <fully-qualified-table-name> with the fully qualified name of the materialized view, including the catalog and schema.
Access the Event Log: The event log for a DLT pipeline can be accessed programmatically. Here is an example query to check if a DLT has run a maintenance task:
SELECT
timestamp,
event_type,
details:maintenance_progress.state,
message,
id,
origin.pipeline_name,
origin.cluster_id,
origin.maintenance_id
FROM delta.`dbfs:/pipelines/<pipeline-id>/system/events`
WHERE event_type = 'maintenance_progress';
Replace <pipeline-id> with the actual pipeline ID.