11-15-2021 05:49 AM
I have a simple SparkSQL Select statement
offers_df = (spark.sql("""
SELECT *
FROM delta.`{}`
""".format(TABLE_LOCATION)))
It runs under DBR 9.0 and previous.
When changing the DBR to 9.1 ...10.1
I get an exception
org.apache.spark.SparkException: Unable to fetch tables of db delta at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1353)
spark_conf of the cluster
"spark_conf": {
"spark.hadoop.fs.s3a.canned.acl": "BucketOwnerFullControl",
"spark.databricks.hive.metastore.glueCatalog.enabled": "true",
"spark.hadoop.fs.s3a.acl.default": "BucketOwnerFullControl"
},
when removing the glueCatalog entry, the Select statement works.
Question:
How can the AWS Glue Catalogue be used in DBR 9.1 .. 10.1 ?
How to configure it in spark_conf?
01-28-2022 12:59 AM
Retest in DBR 10.3 beta SUCCESSFUL.
Problem does not occur in DBR 10.3 beta
11-15-2021 06:35 AM
Hi @ Axel_IW! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
11-29-2021 11:50 AM
Hi @Axel Schwanke ,
I just did a comparison between DBR 9.1 and DBR 10.1 and both have the same library version for Glue.
DBR 9.1 https://docs.databricks.com/release-notes/runtime/9.1.html
DBR 10.1 https://docs.databricks.com/release-notes/runtime/10.1.html
Are you using a high concurrency or standard cluster?
11-29-2021 11:30 PM
No, just a standard cluster.
Everythings works in DBR 9.0, but problem in 9.1 or later
11-30-2021 01:39 AM
Hai @Axel Schwanke We currently have an engineering ticket raised to resolve this issue, we are contemplating the issue is caused because of [SPARK-35531] in the latest release of 10.x
Meanwhile, please continue using the DBR 9.x
11-30-2021 04:37 AM
thanks for the info.
DBR 9.1 was the first version, that had this problem.
12-29-2021 06:45 AM
Retest on 2021-12-29
Suddenly working with DBR 9.1, reason unclear (no change to environment, cluster, notebook, etc.)
Problem persists in DBR 10.0 - 10.2
12-30-2021 09:22 AM
@Axel Schwanke - Thank you for the update. I'll pass the information along. Thanks in advance for your patience.
01-28-2022 12:59 AM
Retest in DBR 10.3 beta SUCCESSFUL.
Problem does not occur in DBR 10.3 beta
01-28-2022 01:04 AM
Hi @Axel Schwanke , Awesome!
Would you like to mark your answer as the best?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.