delta table grouping by key which is not partitioned by is very slow
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-02-2023 04:15 PM
I have a big data delta table with timestamp, key and metric(s) columns (e.g. m1, m2, ...).
I often will group by the key (e.g. select max(m1) group by timestamp, key).
I cannot partition by `key` because there are too many values( ~200K).
I have tried to optimize the table with ZORDER on key, but it does not help.
Bottom line, simple queries take several minutes.
Any idea what to do?