Unsupported datatype 'TimestampNTZType' with liquid clustering
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-26-2023 02:31 PM - edited 12-26-2023 02:55 PM
I'm experimenting with liquid clustering and have some questions about compatible types (somewhat similar to Liquid clustering with boolean columns ).
Table created as
CREATE TABLE IF NOT EXISTS <TABLE>
(
_time DOUBLE
, timestamp TIMESTAMP_NTZ
, aid STRING
, aip STRING
, cid STRING
, TargetProcessId BIGINT
)
USING delta CLUSTER BY (timestamp,aid,TargetProcessId) LOCATION <LOCATION>
Table is filled via .writeStream() so I'm under the impression that data won't be clustered on write. Therefore, I run
OPTIMIZE <TABLE>
which results in (with "spark_version": "14.2.x-cpu-ml-scala2.12"):
IllegalArgumentException: Unsupported datatype 'TimestampNTZType'
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.IllegalArgumentException: Unsupported datatype 'TimestampNTZType'
at com.databricks.sql.io.skipping.Classifier$.getStatsForCol(Classifier.scala:180)
at com.databricks.sql.io.skipping.SimpleClassifier.applyActionToIntersectingBinaryNode(Classifier.scala:424)
I was able to create the table with the CLUSTER BY command so I assumed the column list was acceptable, but this exception indicates otherwise. If TimestampNTZType is an unsupported datatype, where is this documented?
When I run DESCRIBE on the table, it indicates the cluster columns were accepted:
| format | partitionColumns | clusteringColumns | numFiles |
| delta | [] | ["timestamp","aid","TargetProcessId"] | 382 |