โ11-15-2021 05:49 AM
I have a simple SparkSQL Select statement
offers_df = (spark.sql("""
SELECT *
FROM delta.`{}`
""".format(TABLE_LOCATION)))
It runs under DBR 9.0 and previous.
When changing the DBR to 9.1 ...10.1
I get an exception
org.apache.spark.SparkException: Unable to fetch tables of db delta at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1353)
spark_conf of the cluster
"spark_conf": {
"spark.hadoop.fs.s3a.canned.acl": "BucketOwnerFullControl",
"spark.databricks.hive.metastore.glueCatalog.enabled": "true",
"spark.hadoop.fs.s3a.acl.default": "BucketOwnerFullControl"
},
when removing the glueCatalog entry, the Select statement works.
Question:
How can the AWS Glue Catalogue be used in DBR 9.1 .. 10.1 ?
How to configure it in spark_conf?
โ01-28-2022 12:59 AM
Retest in DBR 10.3 beta SUCCESSFUL.
Problem does not occur in DBR 10.3 beta
โ11-15-2021 06:35 AM
Hi @ Axel_IW! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
โ11-29-2021 11:50 AM
Hi @Axel Schwankeโ ,
I just did a comparison between DBR 9.1 and DBR 10.1 and both have the same library version for Glue.
DBR 9.1 https://docs.databricks.com/release-notes/runtime/9.1.html
DBR 10.1 https://docs.databricks.com/release-notes/runtime/10.1.html
Are you using a high concurrency or standard cluster?
โ11-29-2021 11:30 PM
No, just a standard cluster.
Everythings works in DBR 9.0, but problem in 9.1 or later
โ11-30-2021 01:39 AM
Hai @Axel Schwankeโ We currently have an engineering ticket raised to resolve this issue, we are contemplating the issue is caused because of [SPARK-35531] in the latest release of 10.x
Meanwhile, please continue using the DBR 9.x
โ11-30-2021 04:37 AM
thanks for the info.
DBR 9.1 was the first version, that had this problem.
โ12-29-2021 06:45 AM
Retest on 2021-12-29
Suddenly working with DBR 9.1, reason unclear (no change to environment, cluster, notebook, etc.)
Problem persists in DBR 10.0 - 10.2
โ12-30-2021 09:22 AM
@Axel Schwankeโ - Thank you for the update. I'll pass the information along. Thanks in advance for your patience.
โ01-28-2022 12:59 AM
Retest in DBR 10.3 beta SUCCESSFUL.
Problem does not occur in DBR 10.3 beta
โ01-28-2022 01:04 AM
Hi @Axel Schwankeโ , Awesome!
Would you like to mark your answer as the best?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.