https://community.databricks.com/t5/community-articles/liquid-partitioning/m-p/38040#M96 <P>Based on my experience with data partitioning, it often diminishes performance rather than enhancing it. There are exceptions, like when handling tables over 1TB, or when EVERY single query utilizes partition in the WHERE clause - for instance, a PowerBI table constantly filtering on user or date. In other situations, predicting table usage becomes challenging, and partitioning may lead to decreased performance due to issues such as small file problems. Fortunately, databricks now offers a solution to this problem. Your data will automatically adapt to frequently used patterns, thanks to the introduction of liquid partitioning!</P><P>&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ezgif-1-900d8d6a4e.gif" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/2905i37F93F8645EC7855/image-size/large/is-moderation-mode/true?v=v2&amp;px=999" role="button" title="ezgif-1-900d8d6a4e.gif" alt="ezgif-1-900d8d6a4e.gif" /></span></P><P>&nbsp;</P><P>&nbsp;</P> Thu, 20 Jul 2023 13:43:14 GMT Hubert-Dudek 2023-07-20T13:43:14Z https://community.databricks.com/t5/community-articles/liquid-partitioning/m-p/38040#M96 <P>Based on my experience with data partitioning, it often diminishes performance rather than enhancing it. There are exceptions, like when handling tables over 1TB, or when EVERY single query utilizes partition in the WHERE clause - for instance, a PowerBI table constantly filtering on user or date. In other situations, predicting table usage becomes challenging, and partitioning may lead to decreased performance due to issues such as small file problems. Fortunately, databricks now offers a solution to this problem. Your data will automatically adapt to frequently used patterns, thanks to the introduction of liquid partitioning!</P><P>&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ezgif-1-900d8d6a4e.gif" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/2905i37F93F8645EC7855/image-size/large/is-moderation-mode/true?v=v2&amp;px=999" role="button" title="ezgif-1-900d8d6a4e.gif" alt="ezgif-1-900d8d6a4e.gif" /></span></P><P>&nbsp;</P><P>&nbsp;</P> Thu, 20 Jul 2023 13:43:14 GMT https://community.databricks.com/t5/community-articles/liquid-partitioning/m-p/38040#M96 Hubert-Dudek 2023-07-20T13:43:14Z https://community.databricks.com/t5/community-articles/liquid-partitioning/m-p/56892#M97 <P>Thank you for sharing this&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/25346">@Hubert-Dudek</a>&nbsp;!!</P> Wed, 10 Jan 2024 19:59:28 GMT https://community.databricks.com/t5/community-articles/liquid-partitioning/m-p/56892#M97 jose_gonzalez 2024-01-10T19:59:28Z