topic Re: DLT Event Log in Data Engineering

DLT Event Log

kfoster — Tue, 06 Sep 2022 13:12:34 GMT

I am trying to utilize the Event Log DLT is keeping updated, I noticed some of the fields are consistently empty/null.

In the Event Log, located ".../storage/system/events", I see the field "origin" and there are nested fields within which are empty/null. How does the "table_id", "table_name", and "batch_id" become populated?

Re: DLT Event Log

jose_gonzalez — Mon, 03 Oct 2022 16:18:26 GMT

Hi @Kristian Foster,

The following docs will provide more details on the event log schema. Please refer to this link https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#monitor-pipelines-with-the-delta-live-tables-event-log

Re: DLT Event Log

kfoster — Thu, 06 Oct 2022 11:22:27 GMT

Thank you @Jose Gonzalez, the question is more about the nested JSON fields within that structure. The fields are null for "table_id", "table_name", and "batch_id".