cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

vinaykumar
by New Contributor III
  • 3393 Views
  • 4 replies
  • 3 kudos

Reading Iceberg table present in S3 from databricks console using spark given none error .

Hi Team , I am facing issue while reading iceberg table from S3 and getting none error when read the data . below steps I followed .Added Iceberg Spark connector library to your Databricks cluster. 2. Cluster Configuration to Enable Iceberg ...

image image
  • 3393 Views
  • 4 replies
  • 3 kudos
Latest Reply
Ambesh
New Contributor III
  • 3 kudos

Hi @Kaniz I am using Databricks Runtime 10.4 ( Spark 3.2 ), so I have downloaded “iceberg-spark-runtime-3.2_2.12”Also the table exists in the S3 bkt. The error msg is:  java.util.NoSuchElementException: None.getI am also attaching a screenshot for re...

  • 3 kudos
3 More Replies
lrodcon
by New Contributor III
  • 5566 Views
  • 4 replies
  • 4 kudos

Read external iceberg table in a spark dataframe within databricks

I am trying to read an external iceberg database from s3 location using the follwing commanddf_source = (spark.read.format("iceberg")   .load(source_s3_path)   .drop(*source_drop_columns)   .filter(f"{date_column}<='{date_filter}'")   )B...

  • 5566 Views
  • 4 replies
  • 4 kudos
Latest Reply
dynofu
New Contributor II
  • 4 kudos

https://issues.apache.org/jira/browse/SPARK-41344

  • 4 kudos
3 More Replies
youssefmrini
by Honored Contributor III
  • 1227 Views
  • 1 replies
  • 2 kudos
  • 1227 Views
  • 1 replies
  • 2 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 2 kudos

Clone can now be used to create and incrementally update Delta tables that mirror Apache Parquet and Apache Iceberg tables. You can update your source Parquet table and incrementally apply the changes to their cloned Delta table with the clone comman...

  • 2 kudos
samrachmiletter
by New Contributor III
  • 1665 Views
  • 4 replies
  • 5 kudos

Resolved! Is it possible to set order of precedence of spark SQL extensions?

I have the iceberg SQL extension installed, but running commands such as MERGE INTO result in the error pyspark.sql.utils.AnalysisException: MERGE destination only supports Delta sources.this seems to be due to using Delta's MERGE command as opposed ...

  • 1665 Views
  • 4 replies
  • 5 kudos
Latest Reply
samrachmiletter
New Contributor III
  • 5 kudos

This does help. I tried going through the DataFrameReader as well but ran into the same error, so it seems it is indeed not possible. Thank you @Hubert Dudek​!

  • 5 kudos
3 More Replies
WojtekJ
by New Contributor
  • 4186 Views
  • 2 replies
  • 3 kudos

Is it possible to use Iceberg instead of DeltaLake?

Hi.Do you know if it is possible to use Iceberg table format instead DeltaLake?Ideally, I would like to see the tables in Databricks stored as Iceberg and use them as usual in the notebooks.I read that there is also an option to link external metasto...

  • 4186 Views
  • 2 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Wojtek J​ , Here's a thorough comparison of Delta Lake, Iceberg and Hudi.This talk shares the research that we did for the comparison of the key features and designs these table format holds, the maturity of features, such as APIs expose to end u...

  • 3 kudos
1 More Replies
Labels