โ08-28-2023 06:08 AM
Hi,
According to the documentation:
https://docs.databricks.com/en/delta-live-tables/observability.html
"The event_log TVF can be called only by the pipeline owner and a view created over the event_log TVF can be queried only by the pipeline owner. The view cannot be shared with other users."
Can you suggest a workaround?
After searching there is another documentation that says"
https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/event_log
"Only owners of the pipeline, streaming table, or materialized view can view the event log. Create a view and grant users access on the view to allow other users to query the event log."
> CREATE VIEW event_log_raw AS SELECT * FROM event_log(table(my_mv));
> GRANT SELECT ON VIEW event_log_raw TO `user@databricks.com`;
After granting users access on the view I created, the user still can't access the view I created.
Would appreciate your help!
Thanks
โ08-28-2023 07:44 AM
@giladba What is the error the user is getting after granting the permission?
โ08-28-2023 07:47 AM
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException: PERMISSION_DENIED: User does not have permission to access event logs of pipeline 'pipelineid'
โ08-29-2023 03:41 AM - edited โ08-29-2023 09:46 AM
Hi @giladba, The error message "com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException: PERMISSION_DENIED: User does not have permission to access event logs of pipeline '_pipelineid_'" indicates that the user does not have the necessary permissions to access the event logs of the specified pipeline. This could be due to the user not having the correct permissions for the securables in the metastore, which can include catalogs, schemas, tables, views, etc.
โข The user not being registered in the account console to access the Unity Catalog.
โข The user does not have the "SELECT" permission on certain tables.
To resolve this issue, you can validate if the user/service principal has the proper permission to access the event logs of the pipeline.
โข Verify if the user is registered in the account console. If not, register the user.
โข Check if the user has "SELECT" permission on the necessary tables. If not, grant the required permissions using the GRANT SELECT ON <table_name> TO <username>
command.
โ08-29-2023 04:17 AM
Thanks. The user is part of a group that has select on the relevant view and all the parent permissions needed for the Unity Catalog.
โ02-23-2024 03:29 AM
Hi,
I am also facing the same issue, even after following all the steps mentioned, I am not able to query the event logs.
any help will be greatly appreciated.
โ05-22-2024 07:00 AM - edited โ05-22-2024 01:37 PM
Hello,
@Kaniz_Fatma We are using Databricks on Azure but I suspect this issue impacts all cloud providers. Essentially, the event_log Table Value Function is only allowing Pipeline Owner to view logs and this is too restrictive. Despite documentation that states the owner can grant SELECT privileges on a view to other users/groups, this functionality does not work (error is unauthorized as documented above): https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/event_log#usage
The documented approach at the link above likely doesn't work as users would need to also be granted SELECT on the underlying object (Table?). In our scenario, we have a Service Principal running our DLT pipelines and need our Data Engineers (group) to be able to view (troubleshoot, monitor, etc.) the event_log of these pipelines.
Would you please raise this as a bug (if intended functionality is to allow granting select permissions) or as a new feature as the current capability is too restrictive.
Resolving this issue will unlock a lot of value for the Databricks community as the event_log contains highly valuable information.
Thank you.
โ06-11-2024 07:52 AM
Is there already a solution for this?
โ06-13-2024 08:02 AM
As per this documentation, https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/unity-catalog, the issue here is documented as a current Limitation:
The following are limitations when using Unity Catalog with Delta Live Tables:
You cannot use the event_log table valued function in a pipeline or query to access the event logs of multiple pipelines. (hcjp: not directly related to this thread but also delivers value when limitation removed)
You cannot share a view created over the event_log table valued function with other users. (hcjp: limitation associated with this specific issue)
@Kaniz_Fatma - I think the community ask, or at least those on this thread, is to get some visibility into when these limitations will be removed. Is this currently on the backlog and, if not, is there some way we can upvote it?
Thank you.
โ06-14-2024 02:24 AM
Thank you for summarizing the issues @hcjp! Looking forward to the reply of @Kaniz_Fatma
โ07-10-2024 03:42 AM - edited โ07-10-2024 03:43 AM
Hi @larsbbb and @hcjp, You can submit feedback directly to the product team to influence the Databricks product roadmap in the following ways:
To quickly submit feedback about your experience with Databricks, fill out the feedback form in your workspace.
To interactively contribute to the product roadmap, submit a feature request in the Ideas Portal. You can view, comment, and vote up other usersโ requests. You can also monitor the progress of your favorite ideas as the Databricks product team goes through their product planning and development process.
To work around this limitation, a few suggestions are provided:
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group