topic Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1 in Data Engineering

Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Axel_Schwanke — Mon, 15 Nov 2021 13:49:34 GMT

I have a simple SparkSQL Select statement

offers_df = (spark.sql("""
SELECT *
FROM delta.`{}`
""".format(TABLE_LOCATION)))

It runs under DBR 9.0 and previous.

When changing the DBR to 9.1 ...10.1

I get an exception

org.apache.spark.SparkException: Unable to fetch tables of db delta at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1353)

spark_conf of the cluster

    "spark_conf": {
        "spark.hadoop.fs.s3a.canned.acl": "BucketOwnerFullControl",
        "spark.databricks.hive.metastore.glueCatalog.enabled": "true",
        "spark.hadoop.fs.s3a.acl.default": "BucketOwnerFullControl"
    },

when removing the glueCatalog entry, the Select statement works.

Question:

How can the AWS Glue Catalogue be used in DBR 9.1 .. 10.1 ?

How to configure it in spark_conf?

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

jose_gonzalez — Mon, 29 Nov 2021 19:50:02 GMT

Hi @Axel Schwanke ,

I just did a comparison between DBR 9.1 and DBR 10.1 and both have the same library version for Glue.

DBR 9.1 https://docs.databricks.com/release-notes/runtime/9.1.html

DBR 10.1 https://docs.databricks.com/release-notes/runtime/10.1.html

Are you using a high concurrency or standard cluster?

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Axel_Schwanke — Tue, 30 Nov 2021 07:30:45 GMT

No, just a standard cluster.

Everythings works in DBR 9.0, but problem in 9.1 or later

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

User16752242491 — Tue, 30 Nov 2021 09:39:24 GMT

Hai @Axel Schwanke We currently have an engineering ticket raised to resolve this issue, we are contemplating the issue is caused because of [SPARK-35531] in the latest release of 10.x

Meanwhile, please continue using the DBR 9.x

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Axel_Schwanke — Tue, 30 Nov 2021 12:37:10 GMT

thanks for the info.

DBR 9.1 was the first version, that had this problem.

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Axel_Schwanke — Wed, 29 Dec 2021 14:45:04 GMT

Retest on 2021-12-29

Suddenly working with DBR 9.1, reason unclear (no change to environment, cluster, notebook, etc.)

Problem persists in DBR 10.0 - 10.2

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Anonymous — Thu, 30 Dec 2021 17:22:45 GMT

@Axel Schwanke - Thank you for the update. I'll pass the information along. Thanks in advance for your patience.

Re: Issue with AWS Glue metacatalogue and DBR 9.1 ... 10.1

Axel_Schwanke — Fri, 28 Jan 2022 08:59:18 GMT

Retest in DBR 10.3 beta SUCCESSFUL.

Problem does not occur in DBR 10.3 beta