How to use Iceberg SQL Extensions in a notebook?

samanthacr
New Contributor II

‎03-05-2025 01:34 PM

I'm trying to use Iceberg's SQL extensions in my Databricks Notebook, but I get a syntax error. Specifically, I'm trying to run 'ALTER TABLE my_iceberg_table WRITE LOCALLY ORDERED BY timestamp;'. This command is listed as part of Iceberg's SQL extensions. How do I configure my Databricks notebook so I can run these commands?

My current set up is:

  • Databricks 13.3 cluster
  • Iceberg jar 'org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.6.0'
    installed as a library
  • Spark conf contains
    • spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    • spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
    • spark.sql.catalog.local.type hadoop
    • spark.sql.catalog.local.warehouse my_s3_path
0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎03-05-2025 07:57 PM

Hi @samanthacr,

Could you please share the exact syntax error, I would try to reproduce this in my environment.

0 Kudos

samanthacr
New Contributor II
In response to Alberto_Umana

‎03-06-2025 10:02 AM

Screenshot 2025-03-06 at 10.01.19 AM.png

 Yes this is the error I get

0 Kudos

gorkaada_BI
New Contributor II

‎03-20-2025 04:07 AM - edited ‎03-20-2025 04:11 AM

Yes can you explain why cluster config:

spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog.warehouse <s3>
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.spark_catalog.catalog-impl org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.spark_catalog.io-impl org.apache.iceberg.aws.s3.S3FileIO

enabling iceberg extension 1.6.1 and 1.8.1 is not working correctly on Databricks cluster (in my case cluster without unity catalog, shared compute 15.4 LTS)

gorkaada_BI_1-1742468749400.png

How to correctly enable stored procedures in cluster SQL endpoint?

 

0 Kudos

gorkaada_BI
New Contributor II

‎03-20-2025 06:25 AM

val dfh = spark.sql(s"""
CALL glue_catalog.system.create_changelog_view(
  table => '<>table',
  options => map('start-snapshot-id', '$startSnapshotId', 'end-snapshot-id', '$endSnapshotId'),
  changelog_view => table_v
)
""")
lead to 
ParseException: [PROCEDURE_NOT_SUPPORTED] Stored procedure is not supported SQLSTATE: 0A000
0 Kudos