Reading Iceberg table present in S3 from databricks console using spark given none error .

[vinaykumar]

Hi Team ,

I am facing issue while reading iceberg table from S3 and getting none error when read the data .

below steps I followed .

1. Added Iceberg Spark connector library to your Databricks cluster.

2. Cluster Configuration to Enable Iceberg

spark.sql.catalog.spark_catalog.warehouse /icebergpoc/

spark.sql.catalog.spark_catalog.type hadoop

spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog

3.created Spark session and configure it to use the Iceberg Spark connector:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("myApp").getOrCreate()

# Configure the Iceberg Spark catalog

spark.conf.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")

spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")

spark.conf.set("spark.sql.catalog.spark_catalog.uri", "s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/metadata/")

4. Load the Iceberg table as a DataFrame using the spark.read.format()

df = spark.read.format("iceberg").load("s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/")

getting Error .

Iceberg table folder Structure

Can you please look into issue and help me to resolve it .

[Debayan]

Hi, Could you please expand the error and provide the details here.

Please tag @Debayan​ with your next comment so that I will get notified. Thank you!

[Ambesh]

Could someone please post the solution, I am stuck in a similar issue. My Iceberg table is created by a separate spark sql script and now i need to read the Iceberg tables(created outside of databricks) from my Databricks notebook.

[Ambesh]

Hi @Retired_mod 
I am using Databricks Runtime 10.4 ( Spark 3.2 ), so I have downloaded “iceberg-spark-runtime-3.2_2.12”
Also the table exists in the S3 bkt. 
The error msg is:  java.util.NoSuchElementException: None.get
I am also attaching a screenshot for reference. 

[messiah]

Hi @Ambesh @vinaykumar 

Did you get your issue resolved ? I am also in the same situation.

[Manabian]

To use Apache Iceberg via the Hadoop Catalog on Databricks, it was found to work with the following settings:

- Use a Databricks Runtime version of 12.2LTS or earlier. 
- Set the access mode to "No isolation shared" (the mode where Unity Catalog cannot be used).
- Use a library compatible with Java 8 (i.e., an Iceberg library earlier than version 1.6.1).
- Apply the necessary Iceberg-related settings in the Spark configuration.

[Ohad-upriver]

I want to use both unity catalog and iceberg that is in a S3 path.

To use unity catalog I can't use access mode "No isolation shared".

Is there a solution for this?