topic Re: Reading Iceberg table present in S3 from databricks console using spark given none error . in Data Engineering

Reading Iceberg table present in S3 from databricks console using spark given none error .

vinaykumar — Tue, 09 May 2023 12:23:39 GMT

Hi Team ,

I am facing issue while reading iceberg table from S3 and getting none error when read the data .

below steps I followed .

Added Iceberg Spark connector library to your Databricks cluster.

2. Cluster Configuration to Enable Iceberg

spark.sql.catalog.spark_catalog.warehouse /icebergpoc/

spark.sql.catalog.spark_catalog.type hadoop

spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog

3.created Spark session and configure it to use the Iceberg Spark connector:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("myApp").getOrCreate()

# Configure the Iceberg Spark catalog

spark.conf.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")

spark.conf.set("spark.sql.catalog.spark_catalog.type", "hadoop")

spark.conf.set("spark.sql.catalog.spark_catalog.uri", "s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/metadata/")

4. Load the Iceberg table as a DataFrame using the spark.read.format()

df = spark.read.format("iceberg").load("s3://agilent-aws-dev-35-enterprise-analytics/it_share/data/current/IT/HANA/LCA/T_LCA_ZLCA_IB_MASTER/")

getting Error .

Iceberg table folder Structure

Can you please look into issue and help me to resolve it .

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

Debayan — Wed, 10 May 2023 06:31:16 GMT

Hi, Could you please expand the error and provide the details here.

Please tag @Debayan with your next comment so that I will get notified. Thank you!

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

Ambesh — Thu, 04 Jan 2024 22:39:15 GMT

Could someone please post the solution, I am stuck in a similar issue. My Iceberg table is created by a separate spark sql script and now i need to read the Iceberg tables(created outside of databricks) from my Databricks notebook.

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

Ambesh — Fri, 05 Jan 2024 17:13:51 GMT

Hi @Retired_mod
I am using Databricks Runtime 10.4 ( Spark 3.2 ), so I have downloaded “iceberg-spark-runtime-3.2_2.12”
Also the table exists in the S3 bkt.
The error msg is: java.util.NoSuchElementException: None.get
I am also attaching a screenshot for reference.

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

messiah — Thu, 20 Feb 2025 04:28:05 GMT

Hi @Ambesh @vinaykumar

Did you get your issue resolved ? I am also in the same situation.

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

Manabian — Wed, 26 Feb 2025 02:26:35 GMT

To use Apache Iceberg via the Hadoop Catalog on Databricks, it was found to work with the following settings:

- Use a Databricks Runtime version of 12.2LTS or earlier.
- Set the access mode to "No isolation shared" (the mode where Unity Catalog cannot be used).
- Use a library compatible with Java 8 (i.e., an Iceberg library earlier than version 1.6.1).
- Apply the necessary Iceberg-related settings in the Spark configuration.

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

Ohad-upriver — Sun, 09 Mar 2025 14:10:28 GMT

I want to use both unity catalog and iceberg that is in a S3 path.

To use unity catalog I can't use access mode "No isolation shared".

Is there a solution for this?

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

jayKumar — Mon, 07 Apr 2025 20:28:56 GMT

Did anyone find solution to use unity catalog and Iceberg table in databricks?

Re: Reading Iceberg table present in S3 from databricks console using spark given none error .

Zuhaib_Mohd — Mon, 16 Feb 2026 08:44:23 GMT

https://community.databricks.com/t5/technical-blog/secure-migration-of-aws-glue-iceberg-tables-to-databricks-unity/ba-p/148414