Is it possible to set order of precedence of spark SQL extensions?

[samrachmiletter]

I have the iceberg SQL extension installed, but running commands such as MERGE INTO result in the error

pyspark.sql.utils.AnalysisException: MERGE destination only supports Delta sources.

this seems to be due to using Delta's MERGE command as opposed to Iceberg's. Is it possible to force use of the Iceberg extension before Databricks' own?

[Hubert-Dudek · Accepted solution]

@Sam Rachmil-Etter​ , SQL syntax is "Databricks" one so I am afraid it will not work. CONVERT can transform the iceberg parquet into the delta.

Probably in python/scala it could be better as format("iceberg") can be used in DataFrameReader but I have no experience with iceberg on databricks.

[Kaniz]

Hi @Sam Rachmil-Etter​ , Just a friendly follow-up. Do you still need help, or @Hubert Dudek (Customer)​ 's response help you to find the solution? Please let us know.

[samrachmiletter]

This does help. I tried going through the DataFrameReader as well but ran into the same error, so it seems it is indeed not possible. Thank you @Hubert Dudek​!

[Kaniz]

Hi @Sam Rachmil-Etter​ , Would you like to mark @Hubert Dudek​ 's answer as the best?