https://community.databricks.com/t5/data-engineering/what-is-the-performance-impact-of-changing/m-p/4158#M958 <P>I already improved a lot the performances of our ETL (x20 !) but I still want to know where I can improve performances. I seems that tables stats and column indexing slow down a bit writings so I want to decrease <B>dataSkippingNumIndexedCols&nbsp;</B>to match the small columns subset on which filters are actually applied, and columns with low distinct values. I wonder if it's worth doing that because it requires development work and columns order is imposed.</P><P>The documentation says it is worth doing that if your columns contain textual information for example because indexing is inefficient.</P> Fri, 19 May 2023 10:07:11 GMT Thor 2023-05-19T10:07:11Z https://community.databricks.com/t5/data-engineering/what-is-the-performance-impact-of-changing/m-p/4158#M958 <P>I already improved a lot the performances of our ETL (x20 !) but I still want to know where I can improve performances. I seems that tables stats and column indexing slow down a bit writings so I want to decrease <B>dataSkippingNumIndexedCols&nbsp;</B>to match the small columns subset on which filters are actually applied, and columns with low distinct values. I wonder if it's worth doing that because it requires development work and columns order is imposed.</P><P>The documentation says it is worth doing that if your columns contain textual information for example because indexing is inefficient.</P> Fri, 19 May 2023 10:07:11 GMT https://community.databricks.com/t5/data-engineering/what-is-the-performance-impact-of-changing/m-p/4158#M958 Thor 2023-05-19T10:07:11Z