TABLE_REDIRECTION_ERROR in AWS Athena After Databricks Upgrade to 14.3 LTS

Tam
New Contributor III

‎02-08-2024 06:32 PM

I have a Databricks pipeline set up to create Delta tables on AWS S3, using Glue Catalog as the Metastore. I was able to query the Delta table via Athena successfully. However, after upgrading Databricks Cluster from 13.3 LTS to 14.3 LTS, I began encountering TABLE_REDIRECTION_ERROR on AWS Athena.

TABLE_REDIRECTION_ERROR: Table 'awsdatacatalog.<database>.<table>' redirected to 'awsdatacatalog$delta-lake-aws.<database>.<table>', 
but the target table 'awsdatacatalog$delta-lake-aws.<database>.<table>' does not exist.!

Tam_1-1707445843989.png

0 Kudos

krystian
New Contributor II
In response to Retired_mod

‎03-13-2024 06:53 AM

Same problem with Athena and TABLE_REDIRECTION_ERROR

Works on runtime 13.3 but not on 14.3 LTS.

 

CREATE TABLE IF NOT EXISTS tablename (
      myid BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),
      name string,
      age int   
      )
      USING delta
      LOCATION 's3://location'
      TBLPROPERTIES('delta.feature.allowColumnDefaults' = 'supported',delta.targetFileSize='128mb',delta.autoOptimize.autoCompact=true,delta.autoOptimize.optimizeWrite = true,spark.databricks.delta.schema.autoMerge.enabled = true)

 

0 Kudos