Hello everybody,

Another strange issue I have and I would like to confirm me if this is a bug or expected behaviour:

I'm joining a large dataset with a dimension table and as expected DPP is activated.

I was trying to deactivate the feature as it changes the read partitions so I disabled through

spark.sql.optimizer.dynamicPartitionPruning.enabled and spark.databricks.optimizer.dynamicPartitionPruning but I STILL had the dynamic partition prunning.

Finally I discovered that by pushing spark.databricks.optimizer.deltaTableFilesThreshold to a big number I managed to see my sql query not to use DPP.

Is this behavior expected? I would say no given the DPP documentation here

Tested on both DBR 9.1 and 10