Databricks Community

Stokholm · 03-28-2023

Hi Everybody,I have 20 years of data, 600m rows.I have partitioned them on year and month to generated a files size which seems reasonable.(128Mb)All data is queried using timestamp, as all queries needs to filter on the exact hours.So my requirement...

Stokholm · 04-24-2023

Hi Guys, thanks for your advices. I found a solution. We upgrade the Databricks Runtime to 12.2 and now the pushdown of the partitionfilter works. The documentation said that 10.4 would be adequate, but obviously it wasn't enough.

Stokholm · 04-13-2023

Hi @Suteja Kanuri I have tried the EXPLAIN command on several different query types. Both with really simple queries and also more complex and restricted on other columns.None of them apply the partitionfilter when only using the timestamp in the wh...

Stokholm · 04-12-2023

Hi @Suteja Kanuri,I think you misunderstood my post. The Movement to Delta Lake in Azure Storage did not help. It still showed that everything was being scanned when using the timestamp in the where-clause.I did not include the partitioned columns t...

Stokholm · 04-04-2023

Hi @Vidula Khanna and @Suteja Kanuri Thanks you for your suggestions. But i haven't really progressed.I set the loglevel to debug. But i'm not quite sure on where to check the outcome.I didnt find any usable hints, that forces the access plan to us...

Databricks Community

User Stats

User Activity

Pushdown of datetime filter to date partition.

Re: Pushdown of datetime filter to date partition.

Re: Pushdown of datetime filter to date partition.

Re: Pushdown of datetime filter to date partition.

Re: Pushdown of datetime filter to date partition.