Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

[Axel_Schwanke]

I have a simple SparkSQL Select statement

offers_df = (spark.sql("""
SELECT *
FROM delta.`{}`
""".format(TABLE_LOCATION)))

It runs under DBR 9.0 and previous.

When changing the DBR to 9.1 ...10.1

I get an exception

org.apache.spark.SparkException: Unable to fetch tables of db delta at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1353)

spark_conf of the cluster

    "spark_conf": {
        "spark.hadoop.fs.s3a.canned.acl": "BucketOwnerFullControl",
        "spark.databricks.hive.metastore.glueCatalog.enabled": "true",
        "spark.hadoop.fs.s3a.acl.default": "BucketOwnerFullControl"
    },

when removing the glueCatalog entry, the Select statement works.

Question:

How can the AWS Glue Catalogue be used in DBR 9.1 .. 10.1 ?

How to configure it in spark_conf?

[jose_gonzalez]

Hi @Axel Schwanke​ ,

I just did a comparison between DBR 9.1 and DBR 10.1 and both have the same library version for Glue.

DBR 9.1 https://docs.databricks.com/release-notes/runtime/9.1.html

DBR 10.1 https://docs.databricks.com/release-notes/runtime/10.1.html

Are you using a high concurrency or standard cluster?

[Axel_Schwanke]

No, just a standard cluster.

Everythings works in DBR 9.0, but problem in 9.1 or later

[User16752242491]

Hai @Axel Schwanke​  We currently have an engineering ticket raised to resolve this issue, we are contemplating the issue is caused because of [SPARK-35531]  in the latest release of 10.x

Meanwhile, please continue using the DBR 9.x

[Axel_Schwanke]

thanks for the info.

DBR 9.1 was the first version, that had this problem.

[Axel_Schwanke]

Retest on 2021-12-29

Suddenly working with DBR 9.1, reason unclear (no change to environment, cluster, notebook, etc.)

Problem persists in DBR 10.0 - 10.2

[Anonymous]

@Axel Schwanke​ - Thank you for the update. I'll pass the information along. Thanks in advance for your patience.

[Axel_Schwanke · Accepted solution]

Retest in DBR 10.3 beta SUCCESSFUL.

Problem does not occur in DBR 10.3 beta