Handling partition overwrite in Liquid Clustering

Klusener — Tue, 10 Jun 2025 06:27:22 GMT

Hello,

Currently we have delta tables in TBs partitioned by year, month, day. We perform dynamic partition overwrite using partitionOverwriteMode as dynamic to handle rerun/corrections.

With liquid clustering, since explicit partitions are not required, how should we handle overwrite scenarios? Using MERGE is not feasible in our case, as the data consists of large log files with 100–1000 attributes.

Thanks

Re: Handling partition overwrite in Liquid Clustering

Saritha_S — Wed, 11 Jun 2025 16:25:44 GMT

Hi @Klusener

Good day!!

Dynamic partition overwrites only supports selective overwrites for partitioned columns, not for liquid clustering or regular columns.

If you know the exact predicates, use replaceWhere. Note: This is not possible without knowing the predicates

Reference doc: https://docs.databricks.com/aws/en/delta/selective-overwrite#arbitrary-selective-overwrite-with-replacewhere

To support this feature, we have a feature request, and below is the link for reference, which can be tracked in the future.

Reference: https://databricks.aha.io/ideas/ideas/DB-I-10561

There is no ETA for the feature request currently. Our PM team will review the feature request and will prioritize it accordingly.

Please let us know if you need any further assistance or questions on this.

topic Handling partition overwrite in Liquid Clustering in Data Engineering

Handling partition overwrite in Liquid Clustering

Re: Handling partition overwrite in Liquid Clustering