Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2021 08:21 PM
- The ANALYZE Command specifically captures statistics which are relevant for the Cost Based Optimizer to make better decisions.
- The 32 columns of statistics that Delta auto-collects are specifically for data skipping. This is separate from the ANALYZE command
- The reason docs currently say Do not run on Delta tables’ is because Its best to run Analyze on Delta tables after completion of any data update/delete operation and when the data has changed by around 10%. This gives the CBO the best and most up-to-date statistics to work with
- General best practices:
- ANALYZE whenever the data has changed by about 10%
- Make sure when you use ANALYZE, you are specifying the COLUMNS or PARTITIONS you want to collect statistics for. Otherwise, as you have noted, it will re-analyze the entire table
- Here is the syntax: https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-aux-analyze-table....
