Cannot apply liquid clustering via DLT pipeline

TamD
Contributor

‎05-11-2025 04:53 PM

I want to use liquid clustering on a materialised view created via a DLT pipeline, however, there doesn't appear to be a valid way to do this.

Via table properties:

@Dlt.table(
    name="<table name>,
    comment="<table description",
    table_properties={
        "delta.clusterBy": "AUTO",
        ...
    }
)

 The above code produces the error:

Unknown configuration was specified: delta.clusterBy

DELTA_UNKNOWN_CONFIGURATIONUnknown configuration was specified: delta.clusterBy\

Suggestion from Genie:

@Dlt.table(
    name="<table_name>",
    comment="<table description>",
    table_properties={
        "delta.liquidClustering.enabled": "true"
        ...
    }
)
Produces the same error:

Unknown configuration was specified: delta.liquidClustering.enabled

DELTA_UNKNOWN_CONFIGURATION Unknown configuration was specified: delta.liquidClustering.enabled

 
Further suggestion from Genie is to use a CLUSTER BY clause:
# Enable liquid clustering
spark.sql("ALTER TABLE network_banded_usage CLUSTER BY AUTO")

This produces the error:

'${command}' is not supported in spark.sql("...") API in DLT Python. Supported command: ${supportedCommands}.
UNSUPPORTED_SPARK_SQL_COMMAND'${command}' is not supported in spark.sql("...") API in DLT Python. Supported command: ${supportedCommands}.

I think this is a bug.  Has anyone got liquid clustering enabled via DLT?

0 Kudos