DLT Pipeline & Automatic Liquid Clustering Syntax

[HoussemBL]

Hi everyone,

I noticed Databricks recently released the automatic liquid clustering feature, which looks very promising. I'm currently implementing a DLT pipeline and would like to leverage this new functionality.

However, I'm having trouble figuring out the correct syntax to integrate automatic liquid clustering within my DLT pipeline. I've tried the following code, but it doesn't seem to be working as expected.

dlt.create_streaming_table(
        "table_a",
        schema="""   id STRING NOT NULL,
                    description STRING NOT NULL,
                    is_current BOOLEAN NOT NULL,
        """,
        cluster_by=["auto"],
        comment="table a with automatic liquid clustering",
    )

Could someone please provide an example of the correct syntax for using automatic liquid clustering within a Databricks DLT pipeline? Any guidance or best practices would be greatly appreciated!

Thanks in advance!

[notwarte]

Hi!

I think it's worth trying the same syntax, as is shown here: https://docs.databricks.com/aws/en/delta/clustering?language=Python 

[notwarte]

Also: https://community.databricks.com/t5/community-platform-discussions/cluster-by-auto-pyspark/m-p/11531...

[HoussemBL]

Thanks a lot for your reply @notwarte 
I cannot really use the links that you suggest as I am implementing a DLT pipeline. The syntax of DLT Python is different especially when it comes to creating tables.

[aayrm5]

Hey @HoussemBL 

You're correct about DLT not support Auto LC. You can assign any columns in the cluster_by but if you set it to auto, it will throw an error complaining about auto not being present in the list of columns.

Maybe, altering thee table to set/reset the LC is the only option left as of now.

Let me know your thoughts.

Cheers!

[mai_luca]

It works with SQL syntax (using CLUSTER BY AUTO), but not with pyspark.