delta table grouping by key which is not partitioned by is very slow

chanansh
Contributor

I have a big data delta table with timestamp, key and metric(s) columns (e.g. m1, m2, ...).

I often will group by the key (e.g. select max(m1) group by timestamp, key).

I cannot partition by `key` because there are too many values( ~200K).

I have tried to optimize the table with ZORDER on key, but it does not help.

Bottom line, simple queries take several minutes.

Any idea what to do?