cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Axel_Schwanke
Contributor

I have a simple SparkSQL Select statement

offers_df = (spark.sql("""
SELECT *
FROM delta.`{}`
""".format(TABLE_LOCATION)))

It runs under DBR 9.0 and previous.

When changing the DBR to 9.1 ...10.1

I get an exception

org.apache.spark.SparkException: Unable to fetch tables of db delta at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1353)

spark_conf of the cluster

    "spark_conf": {
        "spark.hadoop.fs.s3a.canned.acl": "BucketOwnerFullControl",
        "spark.databricks.hive.metastore.glueCatalog.enabled": "true",
        "spark.hadoop.fs.s3a.acl.default": "BucketOwnerFullControl"
    },

when removing the glueCatalog entry, the Select statement works.

Question:

How can the AWS Glue Catalogue be used in DBR 9.1 .. 10.1 ?

How to configure it in spark_conf?

1 ACCEPTED SOLUTION

Accepted Solutions

Axel_Schwanke
Contributor

Retest in DBR 10.3 beta SUCCESSFUL.

Problem does not occur in DBR 10.3 beta

View solution in original post

9 REPLIES 9

Kaniz
Community Manager
Community Manager

Hi @ Axel_IW! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

jose_gonzalez
Moderator
Moderator

Hi @Axel Schwanke​ ,

I just did a comparison between DBR 9.1 and DBR 10.1 and both have the same library version for Glue.

DBR 9.1 https://docs.databricks.com/release-notes/runtime/9.1.html

DBR 10.1 https://docs.databricks.com/release-notes/runtime/10.1.html

Are you using a high concurrency or standard cluster?

Axel_Schwanke
Contributor

No, just a standard cluster.

Everythings works in DBR 9.0, but problem in 9.1 or later

User16752242491
New Contributor III
New Contributor III

Hai @Axel Schwanke​  We currently have an engineering ticket raised to resolve this issue, we are contemplating the issue is caused because of [SPARK-35531]  in the latest release of 10.x

Meanwhile, please continue using the DBR 9.x

Axel_Schwanke
Contributor

thanks for the info.

DBR 9.1 was the first version, that had this problem.

Axel_Schwanke
Contributor

Retest on 2021-12-29

Suddenly working with DBR 9.1, reason unclear (no change to environment, cluster, notebook, etc.)

Problem persists in DBR 10.0 - 10.2

Anonymous
Not applicable

@Axel Schwanke​ - Thank you for the update. I'll pass the information along. Thanks in advance for your patience.

Axel_Schwanke
Contributor

Retest in DBR 10.3 beta SUCCESSFUL.

Problem does not occur in DBR 10.3 beta

Hi @Axel Schwanke​ , Awesome!

Would you like to mark your answer as the best?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.