topic Re: CLONE not supported on delta table with Liquid Clustering in Data Engineering

CLONE not supported on delta table with Liquid Clustering

seydouHR — Tue, 05 Dec 2023 22:40:27 GMT

Hello all,

We are building a data warehouse on Unity Catalog and we use the SHALLOW CLONE command to allow folks to spin up their own dev environments by light copying the prod tables.

We also started using Liquid Clustering on our feature tables, though we are running into an error when trying to shallow clone these tables. It seems CLONE command is not allowed for tables that have been liquid clustered.

We'd like to know if this was a temporary restriction / is it in the roadmap at some point in the future to allow a cloning operations on tables that have been liquid clustered ?

Exception :

UnsupportedOperationException: CLONE is not supported for Delta table with Liquid clustering.

Runtime : 13.3

Thanks !

Seydou

Re: CLONE not supported on delta table with Liquid Clustering

seydouHR — Thu, 07 Dec 2023 23:15:35 GMT

Thanks Kaniz for your reply. I was able to get it make it work using runtime 14.0.

Regards,

Re: CLONE not supported on delta table with Liquid Clustering

Wolfoflag — Sun, 03 Mar 2024 03:48:42 GMT

Hi @seydouHR @Retired_mod where you able to shallow clone or deep clone the liquid clustering table in runtime 14.0?

Re: CLONE not supported on delta table with Liquid Clustering

seydouHR — Sun, 03 Mar 2024 13:46:27 GMT

Hi @Wolfoflag ,

Yes, whithin Databricks I was able to shallow clone using runtime 14.0.

Also, outside Databricks, it worked when using the databricks sdk with a SQL Warehouse of version 2023.50 or above.

Example with the sdk:

WORKSPACE_CLIENT.statement_execution.execute_statement(

warehouse_id=sql_warehouse_id, statement=statement

)

Where sql_warehouse_id is the id of your sql warehouse and

your statement could be something like :

f"CREATE OR REPLACE TABLE {target_table_full_name} SHALLOW CLONE {source_table_full_name}"

WORKSPACE_CLIENT is the workspace client from databricks sdk (

i.e from databricks.sdk import WorkspaceClient

)