Hi @seydouHR, The SHALLOW CLONE command in Unity Catalog allows you to create lightweight copies of production tables for development purposes. These clones share metadata with the original tables but do not duplicate the underlying data files, which helps save storage space. It’s a handy feature for creating dev environments.
Now, let’s address the issue you’re encountering with Liquid Clustering. Liquid clustering is a powerful feature available in Delta Lake that simplifies data layout decisions and optimizes query performance. It allows you to redefine clustering keys without rewriting existing data, which is beneficial for evolving analytic needs over time.
However, there are some important considerations:
Liquid Clustering Benefits:
- High Cardinality Columns: Liquid clustering is recommended for tables often filtered by high cardinality columns.
- Data Distribution Skew: It’s useful for tables with significant skew in data distribution.
- Fast-Growing Tables: Tables that grow quickly and require maintenance and tuning effort can benefit from liquid clustering.
- Concurrent Writes: If your tables have concurrent write requirements, liquid clustering provides flexibility.
- Changing Access Patterns: Tables where access patterns change over time can leverage liquid clustering.
- Partition Key Optimization: Liquid clustering helps avoid having too many or too few partitions based on typical partition keys.
Enabling Liquid Clustering:
- When creating a table, add the CLUSTER BY phrase to the table creation statement. For example:CREATE TABLE <catalog-name>.<schema-name>.<target-table-name> SHALLOW CLONE <catalog-name>.<schema-name>.<source-table-name>
- Liquid clustering is not compatible with partitioning or ZORDER. The Databricks client manages all layout and optimization operations for data in the table.
- Once enabled, you can run OPTIMIZE jobs to incrementally cluster data.
Important Notes:
- Databricks Runtime 13.3 LTS and above is required to create, write, or optimize Delta tables with liquid clustering enabled.
- Row-level concurrency is supported in Databricks Runtime 13.3 LTS and above for tables with liquid clustering.
- Tables created with liquid clustering enabled have specific Delta table features enabled at creation.
Limitations:
- CLONE Command: Unfortunately, the CLONE command is not supported for Delta tables with liquid clustering. This restriction prevents you from shallow cloning tables that have undergone liquid clustering.
- Readability: Tables with clustering enabled are not readable by Delta Lake clients that do not support all enabled Delta reader protocol table features.
In summary, while liquid clustering offers significant benefits, it currently restricts the use of the CLONE command. Keep an eye on updates from Databricks for any changes in this regard! 🚀🔍
