Re: Reading Iceberg table present in S3 from datab...

vinaykumar · ‎05-09-2023

Hi Team ,

I am facing issue while reading iceberg table from S3 and getting none error when read the data .

below steps I followed .

Added Iceberg Spark connector library to your Databricks cluster.

2. Cluster Configuration to Enable Iceberg

spark.sql.catalog.spark_catalog.warehouse /icebergpoc/

spark.sql.catalog.spark_catalog.type hadoop

spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog

3.created Spark session and configure it to use the Iceberg Spark connector:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("myApp").getOrCreate()

# Configure the Iceberg Spark catalog

spark.conf.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")

spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")

spark.conf.set("spark.sql.catalog.spark_catalog.uri", "s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/metadata/")

4. Load the Iceberg table as a DataFrame using the spark.read.format()

df = spark.read.format("iceberg").load("s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/")

getting Error .

Iceberg table folder Structure

Can you please look into issue and help me to resolve it .

Debayan · ‎05-09-2023

Hi, Could you please expand the error and provide the details here.

Please tag @Debayan with your next comment so that I will get notified. Thank you!

Ambesh · ‎01-04-2024

Could someone please post the solution, I am stuck in a similar issue. My Iceberg table is created by a separate spark sql script and now i need to read the Iceberg tables(created outside of databricks) from my Databricks notebook.

Ambesh · ‎01-05-2024

Hi @Retired_mod
I am using Databricks Runtime 10.4 ( Spark 3.2 ), so I have downloaded “iceberg-spark-runtime-3.2_2.12”
Also the table exists in the S3 bkt.
The error msg is: java.util.NoSuchElementException: None.get
I am also attaching a screenshot for reference.

messiah · ‎02-19-2025

Hi @Ambesh @vinaykumar

Did you get your issue resolved ? I am also in the same situation.

Manabian · ‎02-25-2025

To use Apache Iceberg via the Hadoop Catalog on Databricks, it was found to work with the following settings:

- Use a Databricks Runtime version of 12.2LTS or earlier.
- Set the access mode to "No isolation shared" (the mode where Unity Catalog cannot be used).
- Use a library compatible with Java 8 (i.e., an Iceberg library earlier than version 1.6.1).
- Apply the necessary Iceberg-related settings in the Spark configuration.

Ohad-upriver · ‎03-09-2025

I want to use both unity catalog and iceberg that is in a S3 path.

To use unity catalog I can't use access mode "No isolation shared".

Is there a solution for this?

jayKumar · ‎04-07-2025

Did anyone find solution to use unity catalog and Iceberg table in databricks?

Reading Iceberg table present in S3 from databricks console using spark given none error .