topic Dynamic Partition Pruning override in Data Engineering

Dynamic Partition Pruning override

pantelis_mare — Thu, 11 Nov 2021 16:00:31 GMT

Hello everybody,

Another strange issue I have and I would like to confirm me if this is a bug or expected behaviour:

I'm joining a large dataset with a dimension table and as expected DPP is activated.

I was trying to deactivate the feature as it changes the read partitions so I disabled through

spark.sql.optimizer.dynamicPartitionPruning.enabled and spark.databricks.optimizer.dynamicPartitionPruning but I STILL had the dynamic partition prunning.

Finally I discovered that by pushing spark.databricks.optimizer.deltaTableFilesThreshold to a big number I managed to see my sql query not to use DPP.

Is this behavior expected? I would say no given the DPP documentation here

Tested on both DBR 9.1 and 10

Re: Dynamic Partition Pruning override

Anonymous — Fri, 12 Nov 2021 16:48:55 GMT

Hello there! Thanks for your question. I'd like to give this a bit longer to see what the community comes up with. Otherwise, I'll bump this to the SMEs.

Re: Dynamic Partition Pruning override

pantelis_mare — Mon, 22 Nov 2021 15:45:02 GMT

Hello @Kaniz Fatma

Thank you for taking the time to answer.

The issue in this case was that spark.databricks.optimizer.deltaTableFilesThreshold was activating DPP even if it was formally deactivated by setting all available "enabled" properties to false (see my initial post)