I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.
I have created a test table using the following query:
CREATE TABLE IF NOT EXISTS test_01 (
id STRING,
name STRING,
country STRING
) USING DELTA
PARTITIONED BY (country)
and then attempt to insert some data using this query:
INSERT OVERWRITE TABLE test_01 PARTITION(country) VALUES ('id_01','name_01','country_01'), ('id_02','name_02','country_02')
I'm getting the following error:
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2
My cluster spark configurations are:
spark.driver.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
spark.databricks.optimizer.adaptive.enabled true
spark.databricks.delta.preview.enabled true
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.sources.partitionOverwriteMode dynamic
spark.sql.adaptive.skewJoin.enabled true
spark.databricks.repl.allowedLanguages python,sql
spark.databricks.acl.dfAclsEnabled true
spark.sql.execution.arrow.enabled true
spark.executor.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
spark.databricks.pyspark.enablePy4JSecurity false
With Databricks runtime version 11.3 LTS (spark 3.3.0)
When executing the same queries on a cluster without ACL everything works fine..
Does anyone encountered such issue and have a solution? Thanks!