topic Why can't I query a table from a cluster, but can query from another cluster in the same workspace in Data Engineering

Why can't I query a table from a cluster, but can query from another cluster in the same workspace

jdhao — Fri, 28 Jul 2023 09:07:25 GMT

I have two clusters A, B under the same azure databricks workspace. Under cluster A, inside my notebook, I tried to query a table: `SELECT * FROM some_table LIMIT 5`. It shows some permission errors. Under cluster B, if I run the same sql query, it works without error.

I thought the table under metastore is accessible to all clusters under the workspace, but apparently I am wrong.

Is there anything settings in cluster level that can explain this issue?

Re: Why can't I query a table from a cluster, but can query from another cluster in the same workspa

Lakshay — Fri, 28 Jul 2023 12:30:55 GMT

Check for any spark config or init script differences in the two clusters.

Re: Why can't I query a table from a cluster, but can query from another cluster in the same workspa

jdhao — Fri, 28 Jul 2023 14:31:45 GMT

Hi Lakshay:

I compared the configurations for the two clusters, I can see that for the cluster where I can execute the SQL query, it has more configurations for spark, the key for those additional configuration:

spark.databricks.acl.dfAclsEnabled
spark.databricks.cluster.profile
spark.databricks.repl.allowedLanguages
spark.hadoop.fs.adls.oauth2.access.token.provider.type
spark.hadoop.fs.adls.oauth2.client.id
spark.hadoop.fs.adls.oauth2.credential
spark.hadoop.fs.adls.oauth2.refresh.url
spark.hadoop.fs.azure.account.auth.type.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth.provider.type.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth2.client.endpoint.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth2.client.id.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth2.client.secret.datalakelakeprdwe002.dfs.core.windows.net

Can check which one of these additonal settings affect the behavior?

Re: Why can't I query a table from a cluster, but can query from another cluster in the same workspa

Lakshay — Fri, 28 Jul 2023 15:52:52 GMT

Hi @jdhao ,

I see a lot of configs related to ADLS authentication(except first 3). All these configs will be needed when you want to access the data stored in ADLS.

Re: Why can't I query a table from a cluster, but can query from another cluster in the same workspa

jdhao — Fri, 28 Jul 2023 16:03:11 GMT

Okay, I see, maybe the tables are physically stored in ADLS, but one cluster is lacking those auth info, so we can not query the tables.