Alberto_Umana
Databricks Employee
Databricks Employee

Hi @jeremy98,

To read the Change Data Feed (CDF) logs in materialized views created by a Delta Live Tables (DLT) pipeline, you can follow these steps:

 

Enable Change Data Feed: Ensure that the change data feed is enabled on the base tables of the materialized views. You can enable it using the following SQL command:

ALTER TABLE table1 SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

 

View the Refresh History: You can view the status of refresh operations on a materialized view, including current and past refreshes, by querying the Delta Live Tables event log. Use the following SQL query to retrieve the relevant information:

 

SELECT *

FROM event_log(TABLE(<fully-qualified-table-name>))

WHERE event_type = 'update_progress'

ORDER BY timestamp DESC;

 

Replace <fully-qualified-table-name> with the fully qualified name of the materialized view, including the catalog and schema.

 

Access the Event Log: The event log for a DLT pipeline can be accessed programmatically. Here is an example query to check if a DLT has run a maintenance task:

 

SELECT

  timestamp,

  event_type,

  details:maintenance_progress.state,

  message,

  id,

  origin.pipeline_name,

  origin.cluster_id,

  origin.maintenance_id

FROM delta.`dbfs:/pipelines/<pipeline-id>/system/events`

WHERE event_type = 'maintenance_progress';

 

Replace <pipeline-id> with the actual pipeline ID.