How to use Iceberg SQL Extensions in a notebook?

[samanthacr]

I'm trying to use Iceberg's SQL extensions in my Databricks Notebook, but I get a syntax error. Specifically, I'm trying to run 'ALTER TABLE my_iceberg_table WRITE LOCALLY ORDERED BY timestamp;'. This command is listed as part of Iceberg's SQL extensions. How do I configure my Databricks notebook so I can run these commands?

My current set up is:

• Databricks 13.3 cluster
• Iceberg jar 'org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.6.0'
  installed as a library
• Spark conf contains
  • spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
  • spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog
  • spark.sql.catalog.local.type hadoop
  • spark.sql.catalog.local.warehouse my_s3_path

[Alberto_Umana]

Hi @samanthacr,

Could you please share the exact syntax error, I would try to reproduce this in my environment.

[samanthacr]

 Yes this is the error I get

[gorkaada_BI]

Yes can you explain why cluster config:

spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog.warehouse <s3>
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.spark_catalog.catalog-impl org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.spark_catalog.io-impl org.apache.iceberg.aws.s3.S3FileIO

enabling iceberg extension 1.6.1 and 1.8.1 is not working correctly on Databricks cluster (in my case cluster without unity catalog, shared compute 15.4 LTS)

How to correctly enable stored procedures in cluster SQL endpoint?

[gorkaada_BI]

val dfh = spark.sql(s"""

CALL glue_catalog.system.create_changelog_view(

  table => '<>table',

  options => map('start-snapshot-id', '$startSnapshotId', 'end-snapshot-id', '$endSnapshotId'),

  changelog_view => table_v

)

""")
lead to 
ParseException: [PROCEDURE_NOT_SUPPORTED] Stored procedure is not supported SQLSTATE: 0A000