Best Practice for Updating Data Skipping Statistics for Additional Columns

pooja_bhumandla
Databricks Partner

Hi Community,
I have a scenario where I’ve already calculated delta statistics for the first 32 columns after enabling the dataskipping property. Now, I need to include 10 more frequently used columns that were not part of the original 32.

Goal:
I want a robust way to calculate statistics for the new 10 columns along with the existing 32 columns without unnecessary recomputation or fragmented maintenance.

Here’s the challenge:
1. If I alter the dataskipping property for all 42 columns (32 existing + 10 new) and recompute statistics for all tables, it introduces unnecessary recomputation.

What’s the recommended approach or best practice here?

Appreciate any insights!