java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2 - Delta tables dynamic partition overwrite on Databricks ACL enabled clusters

[nevoezov]

I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.

I have created a test table using the following query:

CREATE TABLE IF NOT EXISTS test_01 (
    id STRING,
    name STRING,
    country STRING
) USING DELTA
PARTITIONED BY (country)

and then attempt to insert some data using this query:

INSERT OVERWRITE TABLE test_01 PARTITION(country) VALUES ('id_01','name_01','country_01'), ('id_02','name_02','country_02')

I'm getting the following error:

java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2

My cluster spark configurations are:

spark.driver.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
spark.databricks.optimizer.adaptive.enabled true
spark.databricks.delta.preview.enabled true
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.sources.partitionOverwriteMode dynamic
spark.sql.adaptive.skewJoin.enabled true
spark.databricks.repl.allowedLanguages python,sql
spark.databricks.acl.dfAclsEnabled true
spark.sql.execution.arrow.enabled true
spark.executor.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
spark.databricks.pyspark.enablePy4JSecurity false

With Databricks runtime version 11.3 LTS (spark 3.3.0)

When executing the same queries on a cluster without ACL everything works fine..

Does anyone encountered such issue and have a solution? Thanks!

[shubham-rai21]

This behaviour is expected on some Databricks Runtime versions when Table ACLs (ACL-enabled clusters) are enabled.

The key indicator is the error:
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2
Your operation is using:
INSERT OVERWRITE TABLE test_01 PARTITION(country)
combined with:
spark.sql.sources.partitionOverwriteMode=dynamic

On ACL-enabled clusters, Databricks performs additional authorization checks before allowing overwrite operations. Dynamic partition overwrite (OverwritePartitionsDynamic) is treated differently from a standard append or full-table overwrite, and in certain DBR versions the permission validation for this operation is not fully supported, resulting in the SecurityException you're seeing.

A few things to check:
Try a full overwrite instead of dynamic partition overwrite
INSERT OVERWRITE TABLE test_01 VALUES (...)
If this succeeds, it confirms the issue is specific to dynamic partition overwrite authorization.
Test on a newer Databricks Runtime

You're running DBR 11.3 LTS. There have been multiple Delta Lake and ACL-related fixes in later runtimes, so it's worth validating on a more recent LTS version.
Check whether the table is in Hive Metastore or Unity Catalog

Authorization is handled differently depending on the governance model.

Review granted privileges
Ensure the user/service principal has both read and write privileges on the target table and underlying schema.
In my experience, when the same code works on a non-ACL cluster but fails on an ACL-enabled cluster with OverwritePartitionsDynamic, the root cause is typically a limitation or permission validation issue related to Table ACL enforcement rather than Delta Lake itself.