- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-01-2025 12:45 AM
Hello,
We’ve started migrating from Azure Databricks (Hive Metastore) to AWS Databricks with Unity Catalog. Our entire codebase was deliberately designed around spark.sql('...') using DML operations (UPDATE, DELETE, MERGE) for two reasons:
In many cases, performing UPDATE through a DataFrame took tens of minutes (e.g. updating 100 rows could take up to 30 minutes), while the same operation via spark.sql completed in seconds.
SQL-based logic significantly improves readability and flexibility, especially when collaborating with analysts who are not proficient in PySpark.
Problem:
After migrating to Unity Catalog, all spark.sql('UPDATE ...') calls on static Delta tables now fail with:
[UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] UpdateTable are not supported in Unity Catalog.What’s worse is that the error explicitly states no alternative or workaround is provided.
Current options we see:
Rewrite all SQL logic to use DataFrame API — which is expensive and undermines the original design and performance.
Stay on Hive Metastore and forgo Unity Catalog — losing key features like audit logging, fine-grained ACLs, lineage, and external catalogs.
Questions:
Is support for UPDATE, DELETE, MERGE on static Delta tables planned in Unity Catalog?
Is there any officially supported way to retain SQL-based DML compatibility under Unity Catalog?
Are there any planned mechanisms for SQL migration to Unity Catalog without completely switching to DataFrame logic?
Thank you!