- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2022 06:40 AM
Hi @Vladimir Ryabtsev ,
Because you are creating a delta table, I think that you are seeing a performance improvement because of Dynamic Partition pruning,
According to the documentation, "Partition pruning can take place at query compilation time when queries include an explicit literal predicate on the partition key column or it can take place at runtime via Dynamic Partition Pruning." Also do read these documentations if it helps. https://www.databricks.com/blog/2020/04/30/faster-sql-queries-on-delta-lake-with-dynamic-file-prunin...
If you want to test it out, turn off the DFP using spark.databricks.optimizer.dynamicFilePruning by setting it to false and check if the performance is still the same.
If not, it would be great if you posted the DAG so that we can take a look at what is happening.
Hope this helps...Cheers.