Reading an Iceberg table with AWS Glue Data Catalog as metastore
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-05-2024 02:22 PM
I have created an Iceberg table using AWS Glue, however whenever I try to read it using a Databricks cluster, I get `java.lang.InstantiationException`. I have tried every combination of Spark configs for my Databricks compute cluster that I can think of based on Databricks, Dremio, AWS, and Iceberg documentation. Most recently I tried
```
spark.databricks.hive.metastore.glueCatalog.enabled true
spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.0
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```
I have also tried including various `spark.sql.catalog.hive_metastore...` configs as mentioned in the Iceberg docs as well, with the same result. Any guidance on the minimal Spark configs necessary (or other suggestions) to allow reading an Iceberg table with AWS Glue Data Catalog as metastore would be greatly appreciated. Thanks!