https://community.databricks.com/t5/data-engineering/how-many-columns-does-delta-engine-collect-stats-on-for-a-delta/m-p/20915#M14166 <P>by default a delta table has stats collected on the first 32 columns. This setting can be configured using the following.</P><PRE><CODE>set spark.databricks.delta.properties.defaults.dataSkippingNumIndexedCols = 3</CODE></PRE><P>However there's a time trade-off to having a large number of columns set for stats collection. You typically want to collect stats on column that are used in filter, where clauses joins and on which you tend to performance aggregations. </P> Thu, 24 Jun 2021 04:22:03 GMT aladda 2021-06-24T04:22:03Z https://community.databricks.com/t5/data-engineering/how-many-columns-does-delta-engine-collect-stats-on-for-a-delta/m-p/20914#M14165 Thu, 24 Jun 2021 04:19:52 GMT https://community.databricks.com/t5/data-engineering/how-many-columns-does-delta-engine-collect-stats-on-for-a-delta/m-p/20914#M14165 aladda 2021-06-24T04:19:52Z https://community.databricks.com/t5/data-engineering/how-many-columns-does-delta-engine-collect-stats-on-for-a-delta/m-p/20915#M14166 <P>by default a delta table has stats collected on the first 32 columns. This setting can be configured using the following.</P><PRE><CODE>set spark.databricks.delta.properties.defaults.dataSkippingNumIndexedCols = 3</CODE></PRE><P>However there's a time trade-off to having a large number of columns set for stats collection. You typically want to collect stats on column that are used in filter, where clauses joins and on which you tend to performance aggregations. </P> Thu, 24 Jun 2021 04:22:03 GMT https://community.databricks.com/t5/data-engineering/how-many-columns-does-delta-engine-collect-stats-on-for-a-delta/m-p/20915#M14166 aladda 2021-06-24T04:22:03Z