COMPUTE DELTA STATISTICS vs COMPUTE STATISTICS - Data Skipping

[Sainath368]

Hi all,

I recently altered the data skipping stats columns on my Delta Lake table to optimize data skipping. Now, I’m wondering about the best practice for updating statistics:

• Is running ANALYZE TABLE <table_name> COMPUTE DELTA STATISTICS sufficient after such changes?

• Or is it necessary to run a full ANALYZE TABLE <table_name> COMPUTE STATISTICS to fully update statistics, especially for newly incoming data?

I want to understand whether COMPUTE DELTA STATISTICS alone keeps the table statistics fully up to date as new data arrives, or if I should regularly run the full COMPUTE STATISTICS command to ensure accuracy.

Thanks in advance for your guidance!

[Advika]

Hello @Sainath368!

Running ANALYZE TABLE <table_name> COMPUTE DELTA STATISTICS is a good practice after modifying data skipping stats columns on a Delta Lake table. However, this command doesn’t update query optimizer stats. For that, you’ll need to run ANALYZE TABLE <table_name> COMPUTE STATISTICS, which collects table and column-level statistics that help the query optimizer generate efficient query plans.

So, the recommended approach will be:

• Use COMPUTE DELTA STATISTICS after changing data skipping columns or table properties.
• Periodically, or after significant data changes, run COMPUTE STATISTICS to maintain accurate optimizer statistics.

If you’re using Unity Catalog-managed tables, consider enabling predictive optimization to automate statistics collection and related maintenance. Just note that this applies only to Unity Catalog-managed tables.