Why can't I query a table from a cluster, but can query from another cluster in the same workspace
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2023 02:07 AM
I thought the table under metastore is accessible to all clusters under the workspace, but apparently I am wrong.
Is there anything settings in cluster level that can explain this issue?
- Labels:
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2023 05:30 AM
Check for any spark config or init script differences in the two clusters.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2023 07:31 AM
Hi Lakshay:
I compared the configurations for the two clusters, I can see that for the cluster where I can execute the SQL query, it has more configurations for spark, the key for those additional configuration:
spark.databricks.acl.dfAclsEnabled
spark.databricks.cluster.profile
spark.databricks.repl.allowedLanguages
spark.hadoop.fs.adls.oauth2.access.token.provider.type
spark.hadoop.fs.adls.oauth2.client.id
spark.hadoop.fs.adls.oauth2.credential
spark.hadoop.fs.adls.oauth2.refresh.url
spark.hadoop.fs.azure.account.auth.type.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth.provider.type.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth2.client.endpoint.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth2.client.id.datalakelakeprdwe002.dfs.core.windows.net
spark.hadoop.fs.azure.account.oauth2.client.secret.datalakelakeprdwe002.dfs.core.windows.net
Can check which one of these additonal settings affect the behavior?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2023 08:52 AM
Hi @jdhao ,
I see a lot of configs related to ADLS authentication(except first 3). All these configs will be needed when you want to access the data stored in ADLS.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-28-2023 09:03 AM
Okay, I see, maybe the tables are physically stored in ADLS, but one cluster is lacking those auth info, so we can not query the tables.