Why is execution too fast?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-26-2022 04:26 PM
I have a table, full scan of which takes ~20 minutes on my cluster. The table has "Time" TIMESTAMP column and "day" DATE column. The latter is computed (manually) as "Time" truncated to day and used for partitioning.
I query the table using predicate based on "Time" ("day" is not included), but it works too fast (~10 s). I expect that partition skipping is not used. EXPLAIN also shows "PartitionFilters: []", so I assume partitioning cannot account for the performance gain. In fact, adding or removing "day" into predicate does not seem to have any performance impact.
How to explain the the query returning result so fast (~10 s)? What could be other mechanisms that could provide such a performance boost?
Table:
CREATE TABLE myschema.mytable (
Time TIMESTAMP,
TagName STRING,
Value DOUBLE,
Quality INT,
day DATE,
isLate BOOLEAN)
USING delta
PARTITIONED BY (day, isLate)Query:
select date_trunc("minute", Time) as time, TagName, avg(Value) as value
from myschema.mytable
where Time between current_timestamp() - interval '3 days' and current_timestamp()
group by date_trunc("minute", Time), TagNameUpdate 1:
The amount of input it shows for the stage is suspiciously small:
- Labels:
-
Date Column
-
Partitioning