Re: Issue with AWS Glue metacatalogue and DBR 9.1 ...

Axel_Schwanke · ‎11-15-2021

I have a simple SparkSQL Select statement

offers_df = (spark.sql("""
SELECT *
FROM delta.`{}`
""".format(TABLE_LOCATION)))

It runs under DBR 9.0 and previous.

When changing the DBR to 9.1 ...10.1

I get an exception

org.apache.spark.SparkException: Unable to fetch tables of db delta at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1353)

spark_conf of the cluster

    "spark_conf": {
        "spark.hadoop.fs.s3a.canned.acl": "BucketOwnerFullControl",
        "spark.databricks.hive.metastore.glueCatalog.enabled": "true",
        "spark.hadoop.fs.s3a.acl.default": "BucketOwnerFullControl"
    },

when removing the glueCatalog entry, the Select statement works.

Question:

How can the AWS Glue Catalogue be used in DBR 9.1 .. 10.1 ?

How to configure it in spark_conf?

jose_gonzalez · ‎11-29-2021

Hi @Axel Schwanke ,

I just did a comparison between DBR 9.1 and DBR 10.1 and both have the same library version for Glue.

DBR 9.1 https://docs.databricks.com/release-notes/runtime/9.1.html

DBR 10.1 https://docs.databricks.com/release-notes/runtime/10.1.html

Are you using a high concurrency or standard cluster?

Axel_Schwanke · ‎11-29-2021

No, just a standard cluster.

Everythings works in DBR 9.0, but problem in 9.1 or later

User16752242491 · ‎11-30-2021

Hai @Axel Schwanke We currently have an engineering ticket raised to resolve this issue, we are contemplating the issue is caused because of [SPARK-35531] in the latest release of 10.x

Meanwhile, please continue using the DBR 9.x

Axel_Schwanke · ‎11-30-2021

thanks for the info.

DBR 9.1 was the first version, that had this problem.

Axel_Schwanke · ‎12-29-2021

Retest on 2021-12-29

Suddenly working with DBR 9.1, reason unclear (no change to environment, cluster, notebook, etc.)

Problem persists in DBR 10.0 - 10.2

Anonymous · ‎12-30-2021

@Axel Schwanke - Thank you for the update. I'll pass the information along. Thanks in advance for your patience.

Axel_Schwanke · ‎01-28-2022

Retest in DBR 10.3 beta SUCCESSFUL.

Problem does not occur in DBR 10.3 beta

View solution in original post

Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1