Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Hi all, I am trying to Read an external Iceberg table. A separate spark sql script creates my Iceberg table and now i need to read the Iceberg tables(created outside of databricks) from my Databricks notebook. Could someone tell me the approach for that. I tried using spark.read.format("iceberg").load("s3://path to my Iceberg data folder") but getting error. Any help would be appreciated
Hi @-werners- I am using Databricks Runtime 10.4 ( Spark 3.2 ), so I have downloaded “iceberg-spark-runtime-3.2_2.12” Also the table exists in the S3 bkt. The error msg is: java.util.NoSuchElementException: None.get I am also attaching a screenshot for reference.
Hi @-werners- the cluster was provisioned with all the requirements as stated in the doc. I also tried with runtime 13.2 and corresponding Iceberg Jar, this time only the error message changed(which is more informative now) but still Databricks is not able to read the Iceberg tables in S3 with catalog as Glue catalog. The error says: AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] as it is not able to read from the Glue catalog. I also provisioned the instance profile for access to Glue and S3 bucket
Hi @Retired_mod yes the iceberg table does not exist in the default catalog because its created externally(outside of Databricks) by a separate spark sql script. The catalog it uses is Glue catalog. The ques is how can i access that external iceberg table from with in my Databricks notebook
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
