Reading Iceberg table present in S3 from databricks console using spark given none error .

[vinaykumar]

Hi Team ,

I am facing issue while reading iceberg table from S3 and getting none error when read the data .

below steps I followed .

1. Added Iceberg Spark connector library to your Databricks cluster.

2. Cluster Configuration to Enable Iceberg

spark.sql.catalog.spark_catalog.warehouse /icebergpoc/

spark.sql.catalog.spark_catalog.type hadoop

spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog

3.created Spark session and configure it to use the Iceberg Spark connector:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("myApp").getOrCreate()

# Configure the Iceberg Spark catalog

spark.conf.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")

spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")

spark.conf.set("spark.sql.catalog.spark_catalog.uri", "s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/metadata/")

4. Load the Iceberg table as a DataFrame using the spark.read.format()

df = spark.read.format("iceberg").load("s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/")

getting Error .

Iceberg table folder Structure

Can you please look into issue and help me to resolve it .

[Debayan]

Hi, Could you please expand the error and provide the details here.

Please tag @Debayan​ with your next comment so that I will get notified. Thank you!

[Ambesh]

Could someone please post the solution, I am stuck in a similar issue. My Iceberg table is created by a separate spark sql script and now i need to read the Iceberg tables(created outside of databricks) from my Databricks notebook.

[Kaniz]

Hi @Ambesh and @vinaykumar,

Here are a few things you could check:

Permissions: Ensure that your Spark session has the necessary permissions to access the S3 bucket where your Iceberg table is stored.

Table Existence: Verify that the Iceberg table exists at the specified location in your S3 bucket.

Iceberg Version: Check the version of Iceberg you’re using. There might be compatibility issues with certain versions of Spark and Iceberg.

Error Message: Please provide the specific error message you’re seeing. This could give more insight into what might be going wrong.

[Ambesh]

Hi @Kaniz 
I am using Databricks Runtime 10.4 ( Spark 3.2 ), so I have downloaded “iceberg-spark-runtime-3.2_2.12”
Also the table exists in the S3 bkt. 
The error msg is:  java.util.NoSuchElementException: None.get
I am also attaching a screenshot for reference.