https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-z-ordering-and-partitioning/m-p/26657#M18680 <P>Partitioning is a way of&nbsp;distributing the data by keys so that you can restrict the amount of data scanned by each query and <A href="https://kb.databricks.com/delta/delta-merge-into.html#how-to-improve-performance-of-delta-lake-merge-into-queries-using-partition-pruning" alt="https://kb.databricks.com/delta/delta-merge-into.html#how-to-improve-performance-of-delta-lake-merge-into-queries-using-partition-pruning" target="_blank">improve performance</A> /<A href="https://docs.databricks.com/delta/concurrency-control.html?_ga=2.46048486.352253067.1624558321-2086263156.1624558321#avoid-conflicts-using-partitioning-and-disjoint-command-conditions" alt="https://docs.databricks.com/delta/concurrency-control.html?_ga=2.46048486.352253067.1624558321-2086263156.1624558321#avoid-conflicts-using-partitioning-and-disjoint-command-conditions" target="_blank"> avoid conflicts </A></P><P></P><P>General rules of thumb for choosing the right partition columns&nbsp;</P><UL><LI>&nbsp;&nbsp;Cardinality of a column should not be very high</LI><LI>&nbsp;&nbsp;Amount of data in each partition should meet a minimum threshold</LI></UL><P></P><P>Now delta supports a feature called <A href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" alt="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" target="_blank">data skipping to speed up queries</A> . </P><P><A href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clustering" alt="https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clustering" target="_blank">Z-odering </A>is a multi-dimensional clustering approach to colocate related information in the same set of files so that databricks data-skipping algorithms can dramatically reduce the amount of data that needs to be read. This works somewhat like secondary indexes in terms of improving query read performance.&nbsp;</P> Thu, 24 Jun 2021 22:02:47 GMT sajith_appukutt 2021-06-24T22:02:47Z https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-z-ordering-and-partitioning/m-p/26656#M18679 Fri, 28 May 2021 19:22:29 GMT https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-z-ordering-and-partitioning/m-p/26656#M18679 User16790091296 2021-05-28T19:22:29Z https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-z-ordering-and-partitioning/m-p/26657#M18680 <P>Partitioning is a way of&nbsp;distributing the data by keys so that you can restrict the amount of data scanned by each query and <A href="https://kb.databricks.com/delta/delta-merge-into.html#how-to-improve-performance-of-delta-lake-merge-into-queries-using-partition-pruning" alt="https://kb.databricks.com/delta/delta-merge-into.html#how-to-improve-performance-of-delta-lake-merge-into-queries-using-partition-pruning" target="_blank">improve performance</A> /<A href="https://docs.databricks.com/delta/concurrency-control.html?_ga=2.46048486.352253067.1624558321-2086263156.1624558321#avoid-conflicts-using-partitioning-and-disjoint-command-conditions" alt="https://docs.databricks.com/delta/concurrency-control.html?_ga=2.46048486.352253067.1624558321-2086263156.1624558321#avoid-conflicts-using-partitioning-and-disjoint-command-conditions" target="_blank"> avoid conflicts </A></P><P></P><P>General rules of thumb for choosing the right partition columns&nbsp;</P><UL><LI>&nbsp;&nbsp;Cardinality of a column should not be very high</LI><LI>&nbsp;&nbsp;Amount of data in each partition should meet a minimum threshold</LI></UL><P></P><P>Now delta supports a feature called <A href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" alt="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" target="_blank">data skipping to speed up queries</A> . </P><P><A href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clustering" alt="https://docs.databricks.com/delta/optimizations/file-mgmt.html#z-ordering-multi-dimensional-clustering" target="_blank">Z-odering </A>is a multi-dimensional clustering approach to colocate related information in the same set of files so that databricks data-skipping algorithms can dramatically reduce the amount of data that needs to be read. This works somewhat like secondary indexes in terms of improving query read performance.&nbsp;</P> Thu, 24 Jun 2021 22:02:47 GMT https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-z-ordering-and-partitioning/m-p/26657#M18680 sajith_appukutt 2021-06-24T22:02:47Z