topic TABLE_REDIRECTION_ERROR in AWS Athena After Databricks Upgrade to 14.3 LTS in Data Engineering

TABLE_REDIRECTION_ERROR in AWS Athena After Databricks Upgrade to 14.3 LTS

Tam — Fri, 09 Feb 2024 02:32:12 GMT

I have a Databricks pipeline set up to create Delta tables on AWS S3, using Glue Catalog as the Metastore. I was able to query the Delta table via Athena successfully. However, after upgrading Databricks Cluster from 13.3 LTS to 14.3 LTS, I began encountering TABLE_REDIRECTION_ERROR on AWS Athena.

TABLE_REDIRECTION_ERROR: Table 'awsdatacatalog.<database>.<table>' redirected to 'awsdatacatalog$delta-lake-aws.<database>.<table>', but the target table 'awsdatacatalog$delta-lake-aws.<database>.<table>' does not exist.!

Re: TABLE_REDIRECTION_ERROR in AWS Athena After Databricks Upgrade to 14.3 LTS

krystian — Wed, 13 Mar 2024 13:53:21 GMT

Same problem with Athena and TABLE_REDIRECTION_ERROR

Works on runtime 13.3 but not on 14.3 LTS.

CREATE TABLE IF NOT EXISTS tablename (

myid BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),

name string,

age int

)

USING delta

LOCATION 's3://location'

TBLPROPERTIES('delta.feature.allowColumnDefaults' = 'supported',delta.targetFileSize='128mb',delta.autoOptimize.autoCompact=true,delta.autoOptimize.optimizeWrite = true,spark.databricks.delta.schema.autoMerge.enabled = true)