DLT Publish event log to metastore

ankit001mittal — Tue, 22 Apr 2025 13:36:39 GMT

Hi Guys,

I am trying to use DLT Publish event log to metastore feature.

and I noticed it creates a table with the logs for each DLT pipelines separately. Does it mean it maintains the separate log table for all the DLT tables ( in our case, we have 1000s of tables through DLT ) ? or is there a way to include all the logs in 1-2 DLT logs table, like we have job and job task run timelines tables in system.lakeflow schema.
Please let me know if my approach is wrong here, and if there is an efficient way to handle this.

Re: DLT Publish event log to metastore

SP_6721 — Wed, 23 Apr 2025 13:11:10 GMT

Hi @ankit001mittal

Yes, you're right, when you enable the "Publish Event Log to Metastore" option for DLT pipelines, Databricks creates a separate event log table for each pipeline. So, if you have thousands of pipelines, you'll see thousands of log tables. There’s no built-in feature at the moment to automatically combine all logs into a single centralised table like the system tables.

Your approach is valid. If you want a unified view of all logs across pipelines, you can create a SQL view that uses UNION ALL across the individual event log tables. Or set up a streaming process that regularly consolidates logs into one Delta table for easier querying and dashboarding.

topic Re: DLT Publish event log to metastore in Data Engineering

DLT Publish event log to metastore

Re: DLT Publish event log to metastore