Unsupported datatype 'TimestampNTZType' with liquid clustering

hukel
Contributor

I'm experimenting with liquid clustering and have some questions about compatible types  (somewhat similar to Liquid clustering with boolean columns ).

Table created as

 

CREATE TABLE IF NOT EXISTS <TABLE>
(  
    _time DOUBLE
  , timestamp TIMESTAMP_NTZ
  , aid STRING
  , aip STRING
  , cid STRING
  , TargetProcessId BIGINT
)
USING delta CLUSTER BY (timestamp,aid,TargetProcessId) LOCATION <LOCATION>

 

Table is filled via .writeStream() so I'm under the impression that data won't be clustered on write.   Therefore, I run

 

OPTIMIZE <TABLE>

 

which results in (with "spark_version": "14.2.x-cpu-ml-scala2.12"):

 

IllegalArgumentException: Unsupported datatype 'TimestampNTZType'
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.IllegalArgumentException: Unsupported datatype 'TimestampNTZType'
	at com.databricks.sql.io.skipping.Classifier$.getStatsForCol(Classifier.scala:180)
	at com.databricks.sql.io.skipping.SimpleClassifier.applyActionToIntersectingBinaryNode(Classifier.scala:424)

 

I was able to create the table with the CLUSTER BY command so I assumed the column list was acceptable,  but this exception indicates otherwise.   If TimestampNTZType is an unsupported datatype,  where is this documented?

When I run DESCRIBE on the table, it indicates the cluster columns were accepted:

formatpartitionColumnsclusteringColumnsnumFiles
delta[]["timestamp","aid","TargetProcessId"]382