topic Re: I've data in s3/Iceberg tables. How to read it using databricks SparkSQL ? in Data Engineering

I've data in s3/Iceberg tables. How to read it using databricks SparkSQL ?

Nalapriya — Tue, 11 Feb 2025 12:22:45 GMT

I tried this method:

df = spark.read.format("iceberg").load("s3-bucket-path")

But got an error: Multiple sources found for iceberg (com.databricks.sql.transaction.tahoe.uniform.sources.IcebergBrowseOnlyDataSource, org.apache.iceberg.spark.source.IcebergSource), please specify the fully qualified class name.

Then updated the code like this:

df = spark.read.format("org.apache.iceberg.spark.source.IcebergSource").load("s3-bucket-path")

Error: The table or view `default_iceberg`.`s3://my-bucket-path`.`` cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and

How to resolve this issue and read data using databricks SparkSQL?

Note: I've installed the spark runtime jar library, configured the cluster with spark requirements and catalog setup as well.

Re: I've data in s3/Iceberg tables. How to read it using databricks SparkSQL ?

Alberto_Umana — Tue, 11 Feb 2025 12:36:41 GMT

Hi @Nalapriya,

What is the DBR version? and library jar version?

Could you try with below code test?

# Set the catalog and schema
spark.sql("USE CATALOG my_catalog")
spark.sql("USE SCHEMA my_schema")

# Read the Iceberg table
df = spark.read.format("org.apache.iceberg.spark.source.IcebergSource").load("s3://my-bucket-path")

Re: I've data in s3/Iceberg tables. How to read it using databricks SparkSQL ?

Nalapriya — Thu, 20 Feb 2025 15:15:49 GMT

Hi @Alberto_Umana,

I tried the steps you've provided but still I'm not able to read data which is in iceberg format. It would be useful if I get any other suggestions.

Re: I've data in s3/Iceberg tables. How to read it using databricks SparkSQL ?

Ohad-upriver — Sun, 09 Mar 2025 15:14:47 GMT

Hi,

When using Unity catalog, I'm unable to create a new catalog that I can use for querying an Iceberg table.

How can I resolve this issue?

The Iceberg table is stored in S3.