ideal_knee
New Contributor III

Thanks for your quick response. I have added the `iceberg-spark-runtime-3.5_2.12-1.7.0.jar` from iceberg.apache.org as a library in my cluster (runtime is "16.0 ML (includes Apache Spark 3.5.0, Scala 2.12)",) and have the following for my Spark config:

 

spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.glue.catalog-impl org.apache.iceberg.aws.glue.GlueCatalog
spark.databricks.hive.metastore.glueCatalog.enabled true
spark.sql.catalog.glue.type glue
spark.sql.catalog.glue.warehouse s3://<my-bucket>/<my-prefix>/
spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.0
spark.master local[*, 4]
spark.sql.catalog.glue.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.databricks.cluster.profile singleNode
spark.sql.catalog.glue org.apache.iceberg.spark.SparkCatalog

 

 When I run the cluster and try `SHOW TABLES IN glue`, it doesn't find a catalog called `glue`. Are there additional steps that need to be done to make the new `glue` catalog available? I have also tried applying the above `spark.sql.catalog...` changes using the existing catalog name `hive_metastore`, but that does not work either. Some more guidance on reading Iceberg tables with AWS Glue Data Catalog as metastore would be appreciated. The page you linked does not mention Iceberg at all. Thanks!