What are the recommendations around collecting stats on long strings in a Delta Table
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 09:25 PM
It is best to avoid collecting stats on long strings. You typically want to collect stats on column that are used in filter, where clauses, joins and on which you tend to performance aggregations - typically numerical values
You can avoid collecting stats on long strings and improve table processing time by either moving string columns outside of the first 32 columns of a delta table on which stats are collected by default
- alter table change column col after col32
Alternatively you can set stats collection columns to a smaller number using the dataSkippingNumIndexedCols setting
- set spark.databricks.delta.properties.defaults.dataSkippingNumIndexedCols = 3
Labels:
- Labels:
-
Delta table
-
Statistics
0 REPLIES 0

