topic My cluster can't access any tables in data catalogs in Data Engineering

My cluster can't access any tables in data catalogs

bamhn — Tue, 17 Jan 2023 02:52:07 GMT

My goal is to have table access control in the data science and engineering workspace. So I enabled access control to my cluster using this config "spark.databricks.acl.dfAclsEnabled": "true" and my cluster is shown as Table ACLs enabled now (shield logo).

But when I tried to see my tables in the data section, it shows an error in the picture below.

Any suggestions for my problem?

Re: My cluster can't access any tables in data catalogs

Hubert-Dudek — Tue, 17 Jan 2023 09:12:16 GMT

It depends also on the type of the cluster https://docs.databricks.com/clusters/create-cluster.html#access-mode

You also need to enable it for workspace https://docs.databricks.com/administration-guide/access-control/table-acl.html#enable-table-acl

and your subscription needs to be premium.

Re: My cluster can't access any tables in data catalogs

LoiLN — Fri, 04 Aug 2023 14:36:02 GMT

Can I ask you a question, where you can get the "spark.databricks.acl.dfAclsEnabled": "true" config spark value. I also try a lot of times to enable ACLs in the cluster in the data science and engineering workspace like you. But I just only see SQL only like official document: Enable Hive metastore table access control on a cluster (legacy) | Databricks on AWS.
Thanks.

Re: My cluster can't access any tables in data catalogs

Karthik_Venu — Wed, 01 May 2024 20:11:05 GMT

Here is my use case:

https://community.databricks.com/t5/data-engineering/structured-streaming-using-delta-as-source-and-delta-as-sink-and/td-p/67825

And I get this error: "py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.Dataset org.apache.spark.sql.streaming.DataStreamReader.table(java.lang.String) is not whitelisted on class class org.apache.spark.sql.streaming.DataStreamReader "

How do we disable table access control for a cluster when the underlying streaming source and sink tables are under unity catalog?

Thanks