Louis_Frolio
Databricks Employee
Databricks Employee

Greetings @yshah , here are some helpful hints/tips/tricks to guide you.

To access table column statistics when checkpoint V2 is enabled, you can follow these guidelines:

  1. Utilize Databricks Runtime 13.3 LTS or Higher: Ensure that you are using Databricks Runtime 13.3 LTS or above, which supports reading and writing tables with v2 checkpoints.

  2. Leverage Automatic Liquid Clustering: With checkpoint V2, you are able to enable automatic liquid clustering for Unity Catalog managed Delta tables. Using the CLUSTER BY AUTO clause allows Databricks to intelligently choose clustering keys based on historical query workloads. This can help optimize performance while accessing column statistics effectively.

  3. Check for Collected Statistics: By default, the first 32 columns in a Delta table have statistics collected, which can be helpful when querying and utilizing those statistics for performance improvements.

  4. Querying Statistical Data: Use SQL commands like DESCRIBE TABLE table_name; or DESCRIBE DETAIL table_name; to retrieve metadata and statistics for each column. These commands work well in environments where v2 checkpoints are used.

  5. Implement Predictive Optimization: For efficient clustering and access to statistics, ensure that predictive optimization is enabled for the table. This allows the system to re-evaluate and update the clustering keys based on changing query patterns over time.

In summary, use the appropriate Databricks Runtime, enable automatic liquid clustering, ensure statistics are collected, and use relevant SQL commands to access column statistics when working with tables using checkpoint V2.

 

Hope this helps, Louis.