<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Best practices and optimization tips for Delta Lake tables in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174880#M56936</link>
    <description>&lt;P&gt;&lt;SPAN&gt;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/249065"&gt;@Khasim_1&lt;/a&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;/P&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;Delta &lt;/SPAN&gt;&lt;SPAN&gt;Lake &lt;/SPAN&gt;&lt;SPAN&gt;optimization &lt;/SPAN&gt;&lt;SPAN&gt;isn’t &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;one-size-fits-all &lt;/SPAN&gt;&lt;SPAN&gt;recipe, let &lt;/SPAN&gt;&lt;SPAN&gt;Databricks &lt;/SPAN&gt;&lt;SPAN&gt;do &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;heavy &lt;/SPAN&gt;&lt;SPAN&gt;lifting &lt;/SPAN&gt;&lt;SPAN&gt;where &lt;/SPAN&gt;&lt;SPAN&gt;possible, &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;save &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;manual &lt;/SPAN&gt;&lt;SPAN&gt;tuning &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;when &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;workload &lt;/SPAN&gt;&lt;SPAN&gt;actually &lt;/SPAN&gt;&lt;SPAN&gt;asks &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;it! &lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":grinning_face_with_smiling_eyes:"&gt;😄&lt;/span&gt;&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;&lt;SPAN&gt;A few practical recommendations based on table type and workload:&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;1. Unity Catalog managed tables: prefer Predictive Optimization&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;For eligible UC managed tables, &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Predictive Optimization (PO)&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; automatically evaluates and runs&amp;nbsp;OPTIMIZE, VACUUM, and ANALYZE when beneficial.&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;SPAN&gt;Generally, no separate scheduled maintenance jobs are needed.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;Verify PO is enabled; maintenance is asynchronous and its serverless compute is billable.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;For modern managed tables, retain Databricks' default automatic file-size tuning settings.&lt;/SPAN&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;2. External or legacy tables: manage maintenance based on workload&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Where PO isn't available or is disabled:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;STRONG&gt;&lt;SPAN&gt;Auto Optimize:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Optimized Writes and Auto Compaction help reduce small files during and after writes. They complement PO and can operate independently.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;&lt;STRONG&gt;OPTIMIZE:&lt;/STRONG&gt; Start daily for heavily written tables or weekly for lower-volume tables, adjust based on file sizes and query performance.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;&lt;SPAN&gt;VACUUM:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;Weekly is a reasonable starting cadence, with the default &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;7-day retention&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; or longer for time travel, long-running operations and streaming consumers.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;&lt;SPAN&gt;ANALYZE:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Refresh statistics after significant changes or when query plans indicate a need.&lt;/SPAN&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;3. Liquid clustering, partitioning or Z-ORDER?&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;For most new Delta tables, prefer &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;liquid clustering (&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;STRONG&gt;CLUSTER BY AUTO&lt;SPAN&gt;&amp;nbsp;where supported)&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;. It adapts better to evolving query patterns and cannot be combined with traditional partitioning or Z-ORDER.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;When does traditional partitioning still make sense?&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Consider it when workloads have explicit operational boundaries, such as daily/monthly partition replacements or independently scheduled writers processing distinct date ranges.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;&lt;STRONG&gt;However&lt;/STRONG&gt;:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;SPAN&gt;Avoid over-partitioning. Databricks generally advises against partitioning below &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;1 TB&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; and recommends at least &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;1 GB per partition&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;Partitioning isn't required for selective overwrites: &lt;/SPAN&gt;&lt;STRONG&gt;REPLACE WHERE&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;and, on supported runtimes, &lt;/SPAN&gt;&lt;SPAN&gt;&lt;STRONG&gt;REPLACE USING&lt;/STRONG&gt; also support unpartitioned tables.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;Separate partition boundaries don't guarantee conflict-free writes. &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Tables with identity columns, for example, don't support concurrent transactions.&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;For existing Z-ORDER tables, continue using it only when query patterns justify it. PO doesn't automatically maintain Z-ORDER.&lt;/SPAN&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;4. Monitor workload behaviour and use AI-assisted recommendations&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Use &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Query Profile, Query Performance Insights, DESCRIBE HISTORY&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;and file-pruning metrics to identify actual bottlenecks before applying optimizations.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Where available, &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Genie Code's Optimize&amp;nbsp;&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;capabilities can help analyze query performance and recommend SQL or compute improvements. Review recommendations and measure the impact before applying changes.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Overall :&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Start with &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;UC managed Delta tables + Predictive Optimization + liquid clustering&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; where supported. Use Auto Optimize and scheduled maintenance where automatic capabilities aren't sufficient, and traditional partitioning only when it provides a measurable operational benefit.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;References:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;P&gt;&lt;A href="https://docs.databricks.com/aws/en/optimizations/predictive-optimization" target="_self"&gt;&lt;SPAN&gt;Predictive Optimization&lt;/SPAN&gt;&lt;/A&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;A style="font-family: inherit; background-color: #ffffff;" href="https://docs.databricks.com/aws/en/delta/clustering" target="_blank" rel="noopener"&gt;&lt;SPAN&gt;Liquid clustering&lt;/SPAN&gt;&lt;/A&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;A style="font-family: inherit; background-color: #ffffff;" href="https://docs.databricks.com/aws/en/sql/user/queries/performance-insights" target="_self"&gt;Databricks Query Performance Insights and Genie Code&lt;/A&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/UL&gt;</description>
    <pubDate>Wed, 28 Oct 2026 23:00:00 GMT</pubDate>
    <dc:creator>data_pulse</dc:creator>
    <dc:date>2026-10-28T23:00:00Z</dc:date>
    <item>
      <title>Best practices and optimization tips for Delta Lake tables</title>
      <link>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174843#M56922</link>
      <description>&lt;P&gt;Hi everyone,&lt;/P&gt;&lt;P&gt;I’m currently working with Delta Lake and wanted to start a discussion around performance tuning, maintenance, and best practices.&lt;/P&gt;&lt;P&gt;I’m interested in hearing how others in the community approach a few key areas:&lt;/P&gt;&lt;P&gt;Maintenance Operations: How often do you run OPTIMIZE (compaction) and VACUUM on your large production Delta tables?&lt;/P&gt;&lt;P&gt;Partitioning vs. Z-Ordering: What guidelines do you follow when choosing between partitioning a table and using Z-Ordering?&lt;/P&gt;&lt;P&gt;Would love to hear your experiences, lessons learned, or tips for keeping Delta tables running efficiently at scale!&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Wed, 28 Oct 2026 11:16:39 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174843#M56922</guid>
      <dc:creator>Khasim_1</dc:creator>
      <dc:date>2026-10-28T11:16:39Z</dc:date>
    </item>
    <item>
      <title>Re: Best practices and optimization tips for Delta Lake tables</title>
      <link>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174876#M56935</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/249065"&gt;@Khasim_1&lt;/a&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Short version for UC managed tables: turn on predictive optimization and let it run OPTIMIZE and VACUUM for you, and use liquid clustering instead of choosing between partitioning and Z-order. Hand-scheduled maintenance and Z-order are now mainly for external tables or older runtimes.&lt;/P&gt;&lt;P&gt;Maintenance:&lt;/P&gt;&lt;P&gt;- Predictive optimization (enabled at account/catalog/schema level) runs OPTIMIZE, VACUUM and ANALYZE on UC managed tables based on actual usage, so there's no schedule to tune: &lt;A href="https://docs.databricks.com/aws/en/optimizations/predictive-optimization" target="_blank"&gt;https://docs.databricks.com/aws/en/optimizations/predictive-optimization&lt;/A&gt;&lt;/P&gt;&lt;P&gt;- If you schedule it yourself: OPTIMIZE daily (or after big loads) on tables with frequent small writes, weekly for the rest. VACUUM weekly with the default 7-day retention - don't go lower unless you're sure no reader, stream or time-travel query needs older versions.&lt;/P&gt;&lt;P&gt;- For streaming/frequent MERGE tables, enable delta.autoOptimize.optimizeWrite and delta.autoOptimize.autoCompact so you get fewer small files in the first place, and turn on deletion vectors to make MERGE/DELETE cheaper.&lt;/P&gt;&lt;P&gt;Partitioning vs. Z-order:&lt;/P&gt;&lt;P&gt;- Default to liquid clustering: CREATE TABLE ... CLUSTER BY (col1, col2) or ALTER TABLE t CLUSTER BY (...) on an existing table, picking the columns you filter/join on most. Keys can change later without rewriting the table, and OPTIMIZE clusters incrementally. CLUSTER BY AUTO lets Databricks pick keys from your query history: &lt;A href="https://docs.databricks.com/aws/en/delta/clustering" target="_blank"&gt;https://docs.databricks.com/aws/en/delta/clustering&lt;/A&gt;&lt;/P&gt;&lt;P&gt;- Only partition if the table is well over 1 TB and you have a low-cardinality column (usually a date) where each partition holds at least ~1 GB. Over-partitioning (high-cardinality columns, tiny partitions) is the most common cause of slow Delta tables.&lt;/P&gt;&lt;P&gt;- Z-order only on existing partitioned tables where you can't move to liquid yet - it can't be combined with liquid clustering, and it rewrites data on every run.&lt;/P&gt;&lt;P&gt;Quick health check on any table: DESCRIBE DETAIL t - if numFiles is high and sizeInBytes / numFiles is far below ~100 MB, it needs compaction or better clustering.&lt;/P&gt;</description>
      <pubDate>Wed, 28 Oct 2026 21:25:40 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174876#M56935</guid>
      <dc:creator>SumeshKashyap</dc:creator>
      <dc:date>2026-10-28T21:25:40Z</dc:date>
    </item>
    <item>
      <title>Re: Best practices and optimization tips for Delta Lake tables</title>
      <link>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174880#M56936</link>
      <description>&lt;P&gt;&lt;SPAN&gt;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/249065"&gt;@Khasim_1&lt;/a&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;/P&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;Delta &lt;/SPAN&gt;&lt;SPAN&gt;Lake &lt;/SPAN&gt;&lt;SPAN&gt;optimization &lt;/SPAN&gt;&lt;SPAN&gt;isn’t &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;one-size-fits-all &lt;/SPAN&gt;&lt;SPAN&gt;recipe, let &lt;/SPAN&gt;&lt;SPAN&gt;Databricks &lt;/SPAN&gt;&lt;SPAN&gt;do &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;heavy &lt;/SPAN&gt;&lt;SPAN&gt;lifting &lt;/SPAN&gt;&lt;SPAN&gt;where &lt;/SPAN&gt;&lt;SPAN&gt;possible, &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;save &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;manual &lt;/SPAN&gt;&lt;SPAN&gt;tuning &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;when &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;workload &lt;/SPAN&gt;&lt;SPAN&gt;actually &lt;/SPAN&gt;&lt;SPAN&gt;asks &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;it! &lt;/SPAN&gt;&lt;SPAN&gt;&lt;span class="lia-unicode-emoji" title=":grinning_face_with_smiling_eyes:"&gt;😄&lt;/span&gt;&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;&lt;SPAN&gt;A few practical recommendations based on table type and workload:&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;1. Unity Catalog managed tables: prefer Predictive Optimization&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;For eligible UC managed tables, &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Predictive Optimization (PO)&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; automatically evaluates and runs&amp;nbsp;OPTIMIZE, VACUUM, and ANALYZE when beneficial.&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;SPAN&gt;Generally, no separate scheduled maintenance jobs are needed.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;Verify PO is enabled; maintenance is asynchronous and its serverless compute is billable.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;For modern managed tables, retain Databricks' default automatic file-size tuning settings.&lt;/SPAN&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;2. External or legacy tables: manage maintenance based on workload&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Where PO isn't available or is disabled:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;STRONG&gt;&lt;SPAN&gt;Auto Optimize:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Optimized Writes and Auto Compaction help reduce small files during and after writes. They complement PO and can operate independently.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;&lt;STRONG&gt;OPTIMIZE:&lt;/STRONG&gt; Start daily for heavily written tables or weekly for lower-volume tables, adjust based on file sizes and query performance.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;&lt;SPAN&gt;VACUUM:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;Weekly is a reasonable starting cadence, with the default &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;7-day retention&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; or longer for time travel, long-running operations and streaming consumers.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;&lt;SPAN&gt;ANALYZE:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Refresh statistics after significant changes or when query plans indicate a need.&lt;/SPAN&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;3. Liquid clustering, partitioning or Z-ORDER?&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;For most new Delta tables, prefer &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;liquid clustering (&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;STRONG&gt;CLUSTER BY AUTO&lt;SPAN&gt;&amp;nbsp;where supported)&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;. It adapts better to evolving query patterns and cannot be combined with traditional partitioning or Z-ORDER.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;When does traditional partitioning still make sense?&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Consider it when workloads have explicit operational boundaries, such as daily/monthly partition replacements or independently scheduled writers processing distinct date ranges.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;&lt;STRONG&gt;However&lt;/STRONG&gt;:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;SPAN&gt;Avoid over-partitioning. Databricks generally advises against partitioning below &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;1 TB&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; and recommends at least &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;1 GB per partition&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;Partitioning isn't required for selective overwrites: &lt;/SPAN&gt;&lt;STRONG&gt;REPLACE WHERE&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;and, on supported runtimes, &lt;/SPAN&gt;&lt;SPAN&gt;&lt;STRONG&gt;REPLACE USING&lt;/STRONG&gt; also support unpartitioned tables.&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;Separate partition boundaries don't guarantee conflict-free writes. &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Tables with identity columns, for example, don't support concurrent transactions.&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/LI&gt;&lt;LI&gt;&lt;SPAN&gt;For existing Z-ORDER tables, continue using it only when query patterns justify it. PO doesn't automatically maintain Z-ORDER.&lt;/SPAN&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;4. Monitor workload behaviour and use AI-assisted recommendations&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Use &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Query Profile, Query Performance Insights, DESCRIBE HISTORY&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;and file-pruning metrics to identify actual bottlenecks before applying optimizations.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Where available, &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;Genie Code's Optimize&amp;nbsp;&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt;capabilities can help analyze query performance and recommend SQL or compute improvements. Review recommendations and measure the impact before applying changes.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Overall :&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Start with &lt;/SPAN&gt;&lt;STRONG&gt;&lt;SPAN&gt;UC managed Delta tables + Predictive Optimization + liquid clustering&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; where supported. Use Auto Optimize and scheduled maintenance where automatic capabilities aren't sufficient, and traditional partitioning only when it provides a measurable operational benefit.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;References:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;P&gt;&lt;A href="https://docs.databricks.com/aws/en/optimizations/predictive-optimization" target="_self"&gt;&lt;SPAN&gt;Predictive Optimization&lt;/SPAN&gt;&lt;/A&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;A style="font-family: inherit; background-color: #ffffff;" href="https://docs.databricks.com/aws/en/delta/clustering" target="_blank" rel="noopener"&gt;&lt;SPAN&gt;Liquid clustering&lt;/SPAN&gt;&lt;/A&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;A style="font-family: inherit; background-color: #ffffff;" href="https://docs.databricks.com/aws/en/sql/user/queries/performance-insights" target="_self"&gt;Databricks Query Performance Insights and Genie Code&lt;/A&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/UL&gt;</description>
      <pubDate>Wed, 28 Oct 2026 23:00:00 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174880#M56936</guid>
      <dc:creator>data_pulse</dc:creator>
      <dc:date>2026-10-28T23:00:00Z</dc:date>
    </item>
    <item>
      <title>Re: Best practices and optimization tips for Delta Lake tables</title>
      <link>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174890#M56941</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/249065"&gt;@Khasim_1&lt;/a&gt;&amp;nbsp; &amp;nbsp;&lt;/P&gt;&lt;H2&gt;&lt;SPAN&gt;Maintenance &lt;/SPAN&gt;&lt;SPAN&gt;operations: &lt;/SPAN&gt;&lt;SPAN&gt;OPTIMIZE &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;VACUUM&lt;/SPAN&gt;&lt;/H2&gt;&lt;P&gt;&lt;SPAN&gt;The &lt;/SPAN&gt;&lt;SPAN&gt;frequency &lt;/SPAN&gt;&lt;SPAN&gt;of &lt;/SPAN&gt;&lt;SPAN&gt;maintenance &lt;/SPAN&gt;&lt;SPAN&gt;operations &lt;/SPAN&gt;&lt;SPAN&gt;should &lt;/SPAN&gt;&lt;SPAN&gt;depend &lt;/SPAN&gt;&lt;SPAN&gt;on &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;table’s &lt;/SPAN&gt;&lt;SPAN&gt;ingestion &lt;/SPAN&gt;&lt;SPAN&gt;pattern, &lt;/SPAN&gt;&lt;SPAN&gt;update/delete &lt;/SPAN&gt;&lt;SPAN&gt;volume, &lt;/SPAN&gt;&lt;SPAN&gt;file &lt;/SPAN&gt;&lt;SPAN&gt;sizes, &lt;/SPAN&gt;&lt;SPAN&gt;query &lt;/SPAN&gt;&lt;SPAN&gt;performance, &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;storage &lt;/SPAN&gt;&lt;SPAN&gt;cost. &lt;/SPAN&gt;&lt;SPAN&gt;Avoid &lt;/SPAN&gt;&lt;SPAN&gt;running &lt;/SPAN&gt;&lt;SPAN&gt;them &lt;/SPAN&gt;&lt;SPAN&gt;on &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;fixed &lt;/SPAN&gt;&lt;SPAN&gt;schedule &lt;/SPAN&gt;&lt;SPAN&gt;without &lt;/SPAN&gt;&lt;SPAN&gt;checking &lt;/SPAN&gt;&lt;SPAN&gt;whether &lt;/SPAN&gt;&lt;SPAN&gt;they &lt;/SPAN&gt;&lt;SPAN&gt;are &lt;/SPAN&gt;&lt;SPAN&gt;needed.&lt;/SPAN&gt;&lt;/P&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;SPAN&gt;Operation&lt;/SPAN&gt;&lt;SPAN&gt;Recommended &lt;/SPAN&gt;&lt;SPAN&gt;approach&lt;/SPAN&gt;&lt;SPAN&gt;Why&lt;/SPAN&gt; &lt;TABLE&gt;&lt;TBODY&gt;&lt;TR&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;OPTIMIZE&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Daily &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;frequently &lt;/SPAN&gt;&lt;SPAN&gt;updated &lt;/SPAN&gt;&lt;SPAN&gt;tables, &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;weekly &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;stable &lt;/SPAN&gt;&lt;SPAN&gt;tables, &lt;/SPAN&gt;&lt;SPAN&gt;as &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;starting &lt;/SPAN&gt;&lt;SPAN&gt;point&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Compacts &lt;/SPAN&gt;&lt;SPAN&gt;small &lt;/SPAN&gt;&lt;SPAN&gt;files &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;improves &lt;/SPAN&gt;&lt;SPAN&gt;read &lt;/SPAN&gt;&lt;SPAN&gt;performance&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;/TR&gt;&lt;TR&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;VACUUM&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Often &lt;/SPAN&gt;&lt;SPAN&gt;weekly, &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;according &lt;/SPAN&gt;&lt;SPAN&gt;to &lt;/SPAN&gt;&lt;SPAN&gt;retention &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;compliance &lt;/SPAN&gt;&lt;SPAN&gt;requirements&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Removes &lt;/SPAN&gt;&lt;SPAN&gt;eligible &lt;/SPAN&gt;&lt;SPAN&gt;obsolete &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;files &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;reduces &lt;/SPAN&gt;&lt;SPAN&gt;storage &lt;/SPAN&gt;&lt;SPAN&gt;usage&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;/TR&gt;&lt;TR&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;ANALYZE &lt;/SPAN&gt;&lt;SPAN&gt;TABLE&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;After &lt;/SPAN&gt;&lt;SPAN&gt;significant &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;changes, &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;when &lt;/SPAN&gt;&lt;SPAN&gt;query &lt;/SPAN&gt;&lt;SPAN&gt;plans &lt;/SPAN&gt;&lt;SPAN&gt;indicate &lt;/SPAN&gt;&lt;SPAN&gt;stale &lt;/SPAN&gt;&lt;SPAN&gt;statistics&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Helps &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;optimizer &lt;/SPAN&gt;&lt;SPAN&gt;make &lt;/SPAN&gt;&lt;SPAN&gt;better &lt;/SPAN&gt;&lt;SPAN&gt;decisions&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;/TR&gt;&lt;TR&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Liquid &lt;/SPAN&gt;&lt;SPAN&gt;clustering &lt;/SPAN&gt;&lt;SPAN&gt;/ &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;skipping&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Evaluate &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;tables &lt;/SPAN&gt;&lt;SPAN&gt;with &lt;/SPAN&gt;&lt;SPAN&gt;large &lt;/SPAN&gt;&lt;SPAN&gt;volumes &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;selective &lt;/SPAN&gt;&lt;SPAN&gt;filters&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;TD&gt;&lt;P&gt;&lt;SPAN&gt;Can &lt;/SPAN&gt;&lt;SPAN&gt;improve &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;skipping &lt;/SPAN&gt;&lt;SPAN&gt;without &lt;/SPAN&gt;&lt;SPAN&gt;traditional &lt;/SPAN&gt;&lt;SPAN&gt;partition &lt;/SPAN&gt;&lt;SPAN&gt;management&lt;/SPAN&gt;&lt;/P&gt;&lt;/TD&gt;&lt;/TR&gt;&lt;/TBODY&gt;&lt;/TABLE&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;&lt;SPAN&gt;These &lt;/SPAN&gt;&lt;SPAN&gt;are &lt;/SPAN&gt;&lt;SPAN&gt;starting &lt;/SPAN&gt;&lt;SPAN&gt;points, &lt;/SPAN&gt;&lt;SPAN&gt;not &lt;/SPAN&gt;&lt;SPAN&gt;universal &lt;/SPAN&gt;&lt;SPAN&gt;rules. &lt;/SPAN&gt;&lt;SPAN&gt;Monitor &lt;/SPAN&gt;&lt;SPAN&gt;file &lt;/SPAN&gt;&lt;SPAN&gt;counts, &lt;/SPAN&gt;&lt;SPAN&gt;average &lt;/SPAN&gt;&lt;SPAN&gt;file &lt;/SPAN&gt;&lt;SPAN&gt;sizes, &lt;/SPAN&gt;&lt;SPAN&gt;query &lt;/SPAN&gt;&lt;SPAN&gt;latency, &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;write &lt;/SPAN&gt;&lt;SPAN&gt;amplification &lt;/SPAN&gt;&lt;SPAN&gt;to &lt;/SPAN&gt;&lt;SPAN&gt;decide &lt;/SPAN&gt;&lt;SPAN&gt;when &lt;/SPAN&gt;&lt;SPAN&gt;to &lt;/SPAN&gt;&lt;SPAN&gt;run &lt;/SPAN&gt;&lt;SPAN&gt;maintenance.&lt;/SPAN&gt;&lt;/P&gt;&lt;H3&gt;&lt;SPAN&gt;Example: &lt;/SPAN&gt;&lt;SPAN&gt;Optimize &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;Delta &lt;/SPAN&gt;&lt;SPAN&gt;table&lt;/SPAN&gt;&lt;/H3&gt;&lt;P&gt;&lt;SPAN&gt;For tables queried frequently by customer and date, you can consider Z-Ordering if those columns are commonly used in selective filters:&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&amp;nbsp; &amp;nbsp; optimize sales zorder by (customer_id, order_date);&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Z-Ordering &lt;/SPAN&gt;&lt;SPAN&gt;reorganizes &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;to &lt;/SPAN&gt;&lt;SPAN&gt;improve &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;skipping. &lt;/SPAN&gt;&lt;SPAN&gt;It &lt;/SPAN&gt;&lt;SPAN&gt;consumes &lt;/SPAN&gt;&lt;SPAN&gt;compute &lt;/SPAN&gt;&lt;SPAN&gt;resources, &lt;/SPAN&gt;&lt;SPAN&gt;so &lt;/SPAN&gt;&lt;SPAN&gt;use &lt;/SPAN&gt;&lt;SPAN&gt;it &lt;/SPAN&gt;&lt;SPAN&gt;on &lt;/SPAN&gt;&lt;SPAN&gt;columns &lt;/SPAN&gt;&lt;SPAN&gt;that &lt;/SPAN&gt;&lt;SPAN&gt;meaningfully &lt;/SPAN&gt;&lt;SPAN&gt;reduce &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;amount &lt;/SPAN&gt;&lt;SPAN&gt;of &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;scanned.&lt;/SPAN&gt;&lt;/P&gt;&lt;H3&gt;&lt;SPAN&gt;Example: &lt;/SPAN&gt;&lt;SPAN&gt;Vacuum &lt;/SPAN&gt;&lt;SPAN&gt;obsolete &lt;/SPAN&gt;&lt;SPAN&gt;files&lt;/SPAN&gt;&lt;/H3&gt;&lt;P&gt;&lt;SPAN&gt;&amp;nbsp; VACUUM sales;&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;VACUUM is a data-retention operation, not a substitute for compaction. It permanently deletes eligible obsolete data files, subject to retention rules. The default retention period in standard Delta Lake behavior is seven days. Do not shorten it casually: it can affect time travel, long-running readers, and concurrent operations. Confirm your recovery and compliance requirements before changing it.&lt;/SPAN&gt;&lt;/P&gt;&lt;H2&gt;&lt;SPAN&gt;2. &lt;/SPAN&gt;&lt;SPAN&gt;Partitioning &lt;/SPAN&gt;&lt;SPAN&gt;vs. &lt;/SPAN&gt;&lt;SPAN&gt;Z-Ordering &lt;/SPAN&gt;&lt;SPAN&gt;vs. &lt;/SPAN&gt;&lt;SPAN&gt;Liquid &lt;/SPAN&gt;&lt;SPAN&gt;Clustering&lt;/SPAN&gt;&lt;/H2&gt;&lt;P&gt;&lt;SPAN&gt;The &lt;/SPAN&gt;&lt;SPAN&gt;key &lt;/SPAN&gt;&lt;SPAN&gt;is &lt;/SPAN&gt;&lt;SPAN&gt;to &lt;/SPAN&gt;&lt;SPAN&gt;choose &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;layout &lt;/SPAN&gt;&lt;SPAN&gt;based &lt;/SPAN&gt;&lt;SPAN&gt;on &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;table &lt;/SPAN&gt;&lt;SPAN&gt;size &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;actual &lt;/SPAN&gt;&lt;SPAN&gt;query &lt;/SPAN&gt;&lt;SPAN&gt;patterns, &lt;/SPAN&gt;&lt;SPAN&gt;rather &lt;/SPAN&gt;&lt;SPAN&gt;than &lt;/SPAN&gt;&lt;SPAN&gt;applying &lt;/SPAN&gt;&lt;SPAN&gt;every &lt;/SPAN&gt;&lt;SPAN&gt;optimization &lt;/SPAN&gt;&lt;SPAN&gt;technique &lt;/SPAN&gt;&lt;SPAN&gt;to &lt;/SPAN&gt;&lt;SPAN&gt;every &lt;/SPAN&gt;&lt;SPAN&gt;table.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Partitioning&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Use &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;specific &lt;/SPAN&gt;&lt;SPAN&gt;cases &lt;/SPAN&gt;&lt;SPAN&gt;where &lt;/SPAN&gt;&lt;SPAN&gt;partitions &lt;/SPAN&gt;&lt;SPAN&gt;are &lt;/SPAN&gt;&lt;SPAN&gt;large, &lt;/SPAN&gt;&lt;SPAN&gt;reasonably &lt;/SPAN&gt;&lt;SPAN&gt;low-cardinality, &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;frequently &lt;/SPAN&gt;&lt;SPAN&gt;filtered.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Example: &lt;/SPAN&gt;&lt;SPAN&gt;partitioning &lt;/SPAN&gt;&lt;SPAN&gt;historical &lt;/SPAN&gt;&lt;SPAN&gt;data &lt;/SPAN&gt;&lt;SPAN&gt;by &lt;/SPAN&gt;&lt;SPAN&gt;year &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;month &lt;/SPAN&gt;&lt;SPAN&gt;when &lt;/SPAN&gt;&lt;SPAN&gt;each &lt;/SPAN&gt;&lt;SPAN&gt;partition &lt;/SPAN&gt;&lt;SPAN&gt;contains &lt;/SPAN&gt;&lt;SPAN&gt;substantial &lt;/SPAN&gt;&lt;SPAN&gt;data.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Avoid &lt;/SPAN&gt;&lt;SPAN&gt;partitioning &lt;/SPAN&gt;&lt;SPAN&gt;by &lt;/SPAN&gt;&lt;SPAN&gt;customer_id&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;transaction &lt;/SPAN&gt;&lt;SPAN&gt;ID, &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;high-cardinality &lt;/SPAN&gt;&lt;SPAN&gt;timestamps. &lt;/SPAN&gt;&lt;SPAN&gt;Too &lt;/SPAN&gt;&lt;SPAN&gt;many &lt;/SPAN&gt;&lt;SPAN&gt;small &lt;/SPAN&gt;&lt;SPAN&gt;partitions &lt;/SPAN&gt;&lt;SPAN&gt;create &lt;/SPAN&gt;&lt;SPAN&gt;overhead.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Z-Ordering&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Use &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;existing, &lt;/SPAN&gt;&lt;SPAN&gt;non-liquid-clustered &lt;/SPAN&gt;&lt;SPAN&gt;Delta &lt;/SPAN&gt;&lt;SPAN&gt;tables &lt;/SPAN&gt;&lt;SPAN&gt;when &lt;/SPAN&gt;&lt;SPAN&gt;queries &lt;/SPAN&gt;&lt;SPAN&gt;repeatedly &lt;/SPAN&gt;&lt;SPAN&gt;filter &lt;/SPAN&gt;&lt;SPAN&gt;on &lt;/SPAN&gt;&lt;SPAN&gt;selective &lt;/SPAN&gt;&lt;SPAN&gt;columns.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Example: &lt;/SPAN&gt;&lt;SPAN&gt;customer_id&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;product_id&lt;/SPAN&gt;&lt;SPAN&gt;, &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;order_date&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Requires &lt;/SPAN&gt;&lt;SPAN&gt;OPTIMIZE&lt;/SPAN&gt;&lt;SPAN&gt;. &lt;/SPAN&gt;&lt;SPAN&gt;Its &lt;/SPAN&gt;&lt;SPAN&gt;effectiveness &lt;/SPAN&gt;&lt;SPAN&gt;decreases &lt;/SPAN&gt;&lt;SPAN&gt;as &lt;/SPAN&gt;&lt;SPAN&gt;you &lt;/SPAN&gt;&lt;SPAN&gt;add &lt;/SPAN&gt;&lt;SPAN&gt;more &lt;/SPAN&gt;&lt;SPAN&gt;Z-Order &lt;/SPAN&gt;&lt;SPAN&gt;columns, &lt;/SPAN&gt;&lt;SPAN&gt;so &lt;/SPAN&gt;&lt;SPAN&gt;prioritize &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;most &lt;/SPAN&gt;&lt;SPAN&gt;valuable &lt;/SPAN&gt;&lt;SPAN&gt;filter &lt;/SPAN&gt;&lt;SPAN&gt;columns.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Liquid clustering&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Prefer &lt;/SPAN&gt;&lt;SPAN&gt;this &lt;/SPAN&gt;&lt;SPAN&gt;for &lt;/SPAN&gt;&lt;SPAN&gt;many &lt;/SPAN&gt;&lt;SPAN&gt;new &lt;/SPAN&gt;&lt;SPAN&gt;Databricks &lt;/SPAN&gt;&lt;SPAN&gt;Delta &lt;/SPAN&gt;&lt;SPAN&gt;tables, &lt;/SPAN&gt;&lt;SPAN&gt;particularly &lt;/SPAN&gt;&lt;SPAN&gt;large &lt;/SPAN&gt;&lt;SPAN&gt;tables &lt;/SPAN&gt;&lt;SPAN&gt;with &lt;/SPAN&gt;&lt;SPAN&gt;high-cardinality &lt;/SPAN&gt;&lt;SPAN&gt;filters, &lt;/SPAN&gt;&lt;SPAN&gt;changing &lt;/SPAN&gt;&lt;SPAN&gt;query &lt;/SPAN&gt;&lt;SPAN&gt;patterns, &lt;/SPAN&gt;&lt;SPAN&gt;or &lt;/SPAN&gt;&lt;SPAN&gt;skewed &lt;/SPAN&gt;&lt;SPAN&gt;data.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Example: &lt;/SPAN&gt;&lt;SPAN&gt;clustering &lt;/SPAN&gt;&lt;SPAN&gt;a &lt;/SPAN&gt;&lt;SPAN&gt;sales &lt;/SPAN&gt;&lt;SPAN&gt;table &lt;/SPAN&gt;&lt;SPAN&gt;by &lt;/SPAN&gt;&lt;SPAN&gt;customer_id&lt;/SPAN&gt; &lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;order_date&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Clustering &lt;/SPAN&gt;&lt;SPAN&gt;keys &lt;/SPAN&gt;&lt;SPAN&gt;can &lt;/SPAN&gt;&lt;SPAN&gt;be &lt;/SPAN&gt;&lt;SPAN&gt;changed &lt;/SPAN&gt;&lt;SPAN&gt;as &lt;/SPAN&gt;&lt;SPAN&gt;query &lt;/SPAN&gt;&lt;SPAN&gt;patterns &lt;/SPAN&gt;&lt;SPAN&gt;evolve. &lt;/SPAN&gt;&lt;SPAN&gt;Liquid &lt;/SPAN&gt;&lt;SPAN&gt;clustering &lt;/SPAN&gt;&lt;SPAN&gt;is &lt;/SPAN&gt;&lt;SPAN&gt;incompatible &lt;/SPAN&gt;&lt;SPAN&gt;with &lt;/SPAN&gt;&lt;SPAN&gt;traditional &lt;/SPAN&gt;&lt;SPAN&gt;partitioning &lt;/SPAN&gt;&lt;SPAN&gt;and &lt;/SPAN&gt;&lt;SPAN&gt;Z-Ordering &lt;/SPAN&gt;&lt;SPAN&gt;on &lt;/SPAN&gt;&lt;SPAN&gt;the &lt;/SPAN&gt;&lt;SPAN&gt;same &lt;/SPAN&gt;&lt;SPAN&gt;table.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Hope this solution clarifies all your open questions!&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Thu, 29 Oct 2026 02:29:31 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/best-practices-and-optimization-tips-for-delta-lake-tables/m-p/174890#M56941</guid>
      <dc:creator>krishgarikipati</dc:creator>
      <dc:date>2026-10-29T02:29:31Z</dc:date>
    </item>
  </channel>
</rss>

