COMPUTE DELTA STATISTICS vs COMPUTE STATISTICS - Data Skipping

Sainath368 — Tue, 27 May 2025 11:05:37 GMT

Hi all,

I recently altered the data skipping stats columns on my Delta Lake table to optimize data skipping. Now, I’m wondering about the best practice for updating statistics:

Is running ANALYZE TABLE <table_name> COMPUTE DELTA STATISTICS sufficient after such changes?
Or is it necessary to run a full ANALYZE TABLE <table_name> COMPUTE STATISTICS to fully update statistics, especially for newly incoming data?

I want to understand whether COMPUTE DELTA STATISTICS alone keeps the table statistics fully up to date as new data arrives, or if I should regularly run the full COMPUTE STATISTICS command to ensure accuracy.

Thanks in advance for your guidance!

Re: COMPUTE DELTA STATISTICS vs COMPUTE STATISTICS - Data Skipping

Advika — Tue, 27 May 2025 15:18:37 GMT

Hello @Sainath368!

Running ANALYZE TABLE <table_name> COMPUTE DELTA STATISTICS is a good practice after modifying data skipping stats columns on a Delta Lake table. However, this command doesn’t update query optimizer stats. For that, you’ll need to run ANALYZE TABLE <table_name> COMPUTE STATISTICS, which collects table and column-level statistics that help the query optimizer generate efficient query plans.

So, the recommended approach will be:

Use COMPUTE DELTA STATISTICS after changing data skipping columns or table properties.
Periodically, or after significant data changes, run COMPUTE STATISTICS to maintain accurate optimizer statistics.

If you’re using Unity Catalog-managed tables, consider enabling predictive optimization to automate statistics collection and related maintenance. Just note that this applies only to Unity Catalog-managed tables.

topic Re: COMPUTE DELTA STATISTICS vs COMPUTE STATISTICS - Data Skipping in Data Engineering

COMPUTE DELTA STATISTICS vs COMPUTE STATISTICS - Data Skipping

Re: COMPUTE DELTA STATISTICS vs COMPUTE STATISTICS - Data Skipping