topic Re: access to event_log TVF in Data Engineering

access to event_log TVF

giladba — Mon, 28 Aug 2023 13:08:36 GMT

Hi,

According to the documentation:

https://docs.databricks.com/en/delta-live-tables/observability.html

"The event_log TVF can be called only by the pipeline owner and a view created over the event_log TVF can be queried only by the pipeline owner. The view cannot be shared with other users."

Can you suggest a workaround?

After searching there is another documentation that says"
https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/event_log
"Only owners of the pipeline, streaming table, or materialized view can view the event log. Create a view and grant users access on the view to allow other users to query the event log."

> CREATE VIEW event_log_raw AS SELECT * FROM event_log(table(my_mv));
> GRANT SELECT ON VIEW event_log_raw TO `user@databricks.com`;

After granting users access on the view I created, the user still can't access the view I created.

Would appreciate your help!

Thanks

Re: access to event_log TVF

Atanu — Mon, 28 Aug 2023 14:44:11 GMT

@giladba What is the error the user is getting after granting the permission?

Re: access to event_log TVF

giladba — Mon, 28 Aug 2023 14:47:31 GMT

com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException: PERMISSION_DENIED: User does not have permission to access event logs of pipeline 'pipelineid'

Re: access to event_log TVF

giladba — Tue, 29 Aug 2023 11:17:20 GMT

Thanks. The user is part of a group that has select on the relevant view and all the parent permissions needed for the Unity Catalog.

Re: access to event_log TVF

neha_ayodhya — Fri, 23 Feb 2024 11:29:21 GMT

Hi,
I am also facing the same issue, even after following all the steps mentioned, I am not able to query the event logs.
any help will be greatly appreciated.

Re: access to event_log TVF

hcjp — Wed, 22 May 2024 20:37:45 GMT

Hello,

@Retired_mod We are using Databricks on Azure but I suspect this issue impacts all cloud providers. Essentially, the event_log Table Value Function is only allowing Pipeline Owner to view logs and this is too restrictive. Despite documentation that states the owner can grant SELECT privileges on a view to other users/groups, this functionality does not work (error is unauthorized as documented above): https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/event_log#usage

The documented approach at the link above likely doesn't work as users would need to also be granted SELECT on the underlying object (Table?). In our scenario, we have a Service Principal running our DLT pipelines and need our Data Engineers (group) to be able to view (troubleshoot, monitor, etc.) the event_log of these pipelines.

Would you please raise this as a bug (if intended functionality is to allow granting select permissions) or as a new feature as the current capability is too restrictive.

Resolving this issue will unlock a lot of value for the Databricks community as the event_log contains highly valuable information.

Thank you.

Re: access to event_log TVF

larsbbb — Tue, 11 Jun 2024 14:52:00 GMT

Is there already a solution for this?

Re: access to event_log TVF

hcjp — Thu, 13 Jun 2024 15:02:50 GMT

As per this documentation, https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/unity-catalog, the issue here is documented as a current Limitation:

Limitations

The following are limitations when using Unity Catalog with Delta Live Tables:

You cannot use the event_log table valued function in a pipeline or query to access the event logs of multiple pipelines. (hcjp: not directly related to this thread but also delivers value when limitation removed)
You cannot share a view created over the event_log table valued function with other users. (hcjp: limitation associated with this specific issue)

@Retired_mod - I think the community ask, or at least those on this thread, is to get some visibility into when these limitations will be removed. Is this currently on the backlog and, if not, is there some way we can upvote it?

Thank you.

Re: access to event_log TVF

larsbbb — Fri, 14 Jun 2024 09:24:12 GMT

Thank you for summarizing the issues @hcjp! Looking forward to the reply of @Retired_mod

Re: access to event_log TVF

mkEngineer — Mon, 06 Jan 2025 16:05:54 GMT

@Alberto_Umana Maybe you have some insights regarding this?

Re: access to event_log TVF

Alberto_Umana — Mon, 06 Jan 2025 17:24:49 GMT

Hello folks,

I will check internally if there is a feature request on this so I can upvote.

Re: access to event_log TVF

LakehouseGuy — Sun, 02 Mar 2025 11:58:24 GMT

"In our scenario, we have a Service Principal running our DLT pipelines and need our Data Engineers (group) to be able to view (troubleshoot, monitor, etc.) the event_log of these pipelines."

In virtually every scenario for every customer the above will be the case.

Please provide a work around.

Re: access to event_log TVF

larsbbb — Wed, 12 Mar 2025 08:30:03 GMT

@LakehouseGuy @mkEngineer @hcjp @neha_ayodhya I just saw the following option in dlt pipelines! I haven't testing it yet, but it looks promising.

This also looks new documentation:
https://learn.microsoft.com/en-us/azure/databricks/dlt/observability#query-event-log

Maybe someone from the databricks team can confirm this