- 1958 Views
- 1 replies
- 2 kudos
Hi,I know how filtering a delta table on a partition column is a very powerful time-saving approach, but what if this column appears as a CONCAT in the where-clause?I explain my case: I have a delta table with only one partition column, say called co...
- 1958 Views
- 1 replies
- 2 kudos
Latest Reply
I did myself a test and the answer is no:with a Concat filter, spark sql does not know I am using a partition-based column, so it scan all the table.
- 15420 Views
- 9 replies
- 1 kudos
Hi Everybody,I have 20 years of data, 600m rows.I have partitioned them on year and month to generated a files size which seems reasonable.(128Mb)All data is queried using timestamp, as all queries needs to filter on the exact hours.So my requirement...
- 15420 Views
- 9 replies
- 1 kudos
Latest Reply
Hi Guys, thanks for your advices. I found a solution. We upgrade the Databricks Runtime to 12.2 and now the pushdown of the partitionfilter works. The documentation said that 10.4 would be adequate, but obviously it wasn't enough.
8 More Replies
- 7959 Views
- 1 replies
- 1 kudos
Hi,maybe someone can help me i do want to run very narrow query SELECT *
FROM my_table
WHERE snapshot_date IN('2023-01-06', '2023-01-07')
-- part of the physical plan:
-- Location: PreparedDeltaFileIndex [dbfs:/...]
-- PartitionFilters: [cast(snaps...
- 7959 Views
- 1 replies
- 1 kudos
Latest Reply
No hints on partition pruning afaik.The reason the partitions were not pruned is because the second query generates a completely different plan.To be able to filter the partitions, a join first has to happen. And in this case it means the table has...
