topic Re: How to use Iceberg SQL Extensions in a notebook? in Data Engineering

How to use Iceberg SQL Extensions in a notebook?

samanthacr — Wed, 05 Mar 2025 21:34:55 GMT

I'm trying to use Iceberg's SQL extensions in my Databricks Notebook, but I get a syntax error. Specifically, I'm trying to run 'ALTER TABLE my_iceberg_table WRITE LOCALLY ORDERED BY timestamp;'. This command is listed as part of Iceberg's SQL extensions. How do I configure my Databricks notebook so I can run these commands?

My current set up is:

Databricks 13.3 cluster
Iceberg jar 'org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.6.0'
installed as a library
Spark conf contains
- spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
- spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
- spark.sql.catalog.local.type hadoop
- spark.sql.catalog.local.warehouse my_s3_path

Re: How to use Iceberg SQL Extensions in a notebook?

Alberto_Umana — Thu, 06 Mar 2025 03:57:55 GMT

Hi @samanthacr,

Could you please share the exact syntax error, I would try to reproduce this in my environment.

Re: How to use Iceberg SQL Extensions in a notebook?

samanthacr — Thu, 06 Mar 2025 18:02:20 GMT

Yes this is the error I get

Re: How to use Iceberg SQL Extensions in a notebook?

gorkaada_BI — Thu, 20 Mar 2025 11:11:05 GMT

Yes can you explain why cluster config:

spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog.warehouse <s3>
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.spark_catalog.catalog-impl org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.spark_catalog.io-impl org.apache.iceberg.aws.s3.S3FileIO

enabling iceberg extension 1.6.1 and 1.8.1 is not working correctly on Databricks cluster (in my case cluster without unity catalog, shared compute 15.4 LTS)

How to correctly enable stored procedures in cluster SQL endpoint?

Re: How to use Iceberg SQL Extensions in a notebook?

gorkaada_BI — Thu, 20 Mar 2025 13:25:08 GMT

val dfh = spark.sql(s"""

CALL glue_catalog.system.create_changelog_view(

table => '<>table',

options => map('start-snapshot-id', '$startSnapshotId', 'end-snapshot-id', '$endSnapshotId'),

changelog_view => table_v

)

""")
lead to
ParseException: [PROCEDURE_NOT_SUPPORTED] Stored procedure is not supported SQLSTATE: 0A000