topic Liquid Clustering limitation clustering on write does not support source queries that include filter in Data Engineering

Liquid Clustering limitation clustering on write does not support source queries that include filter

ShivangiB — Fri, 07 Mar 2025 07:23:49 GMT

I have a query :

%sql

insert into ucdata.brz.liquidcluster_table_data

select sum(col1) as col1,col2,sum(col3) as col3 from ucdata.brz.liquidcluster_table_data group by col2

This query I am running with run time version 13.3 and it is still working.

But as per databricks limitation doc it should not work.

Can you please help.

Re: Liquid Clustering limitation clustering on write does not support source queries that include fi

SantoshJoshi — Sun, 09 Mar 2025 00:02:15 GMT

Hi @ShivangiB,

Can you share the definition of your table - ucdata.brz.liquidcluster_table_data?

Curious - why your source and target are the same table? Are you trying to store the different grains of data in the same table?

I think you are referring this article - Use liquid clustering for Delta tables - Azure Databricks | Microsoft Learn

Re: Liquid Clustering limitation clustering on write does not support source queries that include fi

ShivangiB — Mon, 10 Mar 2025 07:16:27 GMT

Hey @SantoshJoshi , I am just trying too test the limitations specified for some case study. And while testing can't see these limitations so wanted to understand.

it is having 3 columns , col1,col2,col3 having random numbers and col2 is cluster by column.

Re: Liquid Clustering limitation clustering on write does not support source queries that include fi

ShivangiB — Wed, 12 Mar 2025 04:11:15 GMT

Hey Team, can you please help on this