DLT Publish event log to metastore

[ankit001mittal]

Hi Guys,

I am trying to use DLT Publish event log to metastore feature.

and I noticed it creates a table with the logs for each DLT pipelines separately. Does it mean it maintains the separate log table for all the DLT tables ( in our case, we have 1000s of tables through DLT ) ? or is there a way to include all the logs in 1-2 DLT logs table, like we have job and job task run timelines tables in system.lakeflow schema. 
Please let me know if my approach is wrong here, and if there is an efficient way to handle this.

[SP_6721]

Hi @ankit001mittal 

Yes, you're right, when you enable the "Publish Event Log to Metastore" option for DLT pipelines, Databricks creates a separate event log table for each pipeline. So, if you have thousands of pipelines, you'll see thousands of log tables. There’s no built-in feature at the moment to automatically combine all logs into a single centralised table like the system tables.

Your approach is valid. If you want a unified view of all logs across pipelines, you can create a SQL view that uses UNION ALL across the individual event log tables. Or set up a streaming process that regularly consolidates logs into one Delta table for easier querying and dashboarding.