topic Re: DLT Pipeline Event Logs in Data Engineering

DLT Pipeline Event Logs

eballinger — Thu, 09 Jan 2025 20:09:37 GMT

There seems to be a issue now with our DLT pipeline event logs. I am not sure if this is a recent bug or not (but they were ok in Dec). But the issue is in dev, qc and prod and we only have a couple days of history logs now visible in the UI.

From what I have read you are supposed to be able to query the event logs directly (outside of the UI) like this:

SELECT * FROM event_log("pipeline_id_here")

But this gives permission denied and from what I can read only the pipeline owner can view this table info directly. So if you run a pipeline as service principal you cant view it and in dev only the dev owner (not me) would be able to from what I can read. So this is not practical at all.

Is this a recent bug or something? Where are the event logs now and how can I view them?

Thanks

Re: DLT Pipeline Event Logs

Walter_C — Thu, 09 Jan 2025 20:15:12 GMT

Can you share the exact error message you are currently receiving?

Re: DLT Pipeline Event Logs

Walter_C — Thu, 09 Jan 2025 20:17:58 GMT

As per doc indeed The event_log TVF can be called only by the pipeline owner and a view created over the event_log TVF can be queried only by the pipeline owner. The view cannot be shared with other users.

When you were able to see the event logs, this pipelines were also implemented in UC?

Re: DLT Pipeline Event Logs

eballinger — Thu, 09 Jan 2025 20:33:30 GMT

These pipelines are in UC yes. Also just for reference, when I try and view the event log history from the UI dropdown I get this message for a past date "Displaying the latest update. Specified update <pipeline_id> not found. Displaying the latest update instead"

Re: DLT Pipeline Event Logs

Walter_C — Thu, 09 Jan 2025 20:49:28 GMT

if you try executing SQL statement from API call using token for the SP does it show you the information

Re: DLT Pipeline Event Logs

eballinger — Fri, 17 Jan 2025 13:26:48 GMT

Just wanted to update this post. It look like there must have been a bug in the user interface. We can see the event logs again now in the pipeline UI. As for the SQL query method, yes it it now understood that only the pipeline owner can query that part. Thanks Walter

Re: DLT Pipeline Event Logs

Walter_C — Fri, 17 Jan 2025 15:03:13 GMT

Great to hear your issue got resolved.