- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 weeks ago
This behaviour is expected on some Databricks Runtime versions when Table ACLs (ACL-enabled clusters) are enabled.
The key indicator is the error:
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2
Your operation is using:
INSERT OVERWRITE TABLE test_01 PARTITION(country)
combined with:
spark.sql.sources.partitionOverwriteMode=dynamic
On ACL-enabled clusters, Databricks performs additional authorization checks before allowing overwrite operations. Dynamic partition overwrite (OverwritePartitionsDynamic) is treated differently from a standard append or full-table overwrite, and in certain DBR versions the permission validation for this operation is not fully supported, resulting in the SecurityException you're seeing.
A few things to check:
Try a full overwrite instead of dynamic partition overwrite
INSERT OVERWRITE TABLE test_01 VALUES (...)
If this succeeds, it confirms the issue is specific to dynamic partition overwrite authorization.
Test on a newer Databricks Runtime
You're running DBR 11.3 LTS. There have been multiple Delta Lake and ACL-related fixes in later runtimes, so it's worth validating on a more recent LTS version.
Check whether the table is in Hive Metastore or Unity Catalog
Authorization is handled differently depending on the governance model.
Review granted privileges
Ensure the user/service principal has both read and write privileges on the target table and underlying schema.
In my experience, when the same code works on a non-ACL cluster but fails on an ACL-enabled cluster with OverwritePartitionsDynamic, the root cause is typically a limitation or permission validation issue related to Table ACL enforcement rather than Delta Lake itself.