https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23143#M15943 <P>good question <span class="lia-unicode-emoji" title=":winking_face:">😉</span> I was going to mention this.</P><P></P><P>You can still use external tables. Those tables will be stored outside the main (root) metastore bucket/container.</P><P></P><P><A href="https://docs.databricks.com/data-governance/unity-catalog/create-tables.html" alt="https://docs.databricks.com/data-governance/unity-catalog/create-tables.html" target="_blank">https://docs.databricks.com/data-governance/unity-catalog/create-tables.html</A></P><P></P><P><A href="https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html?&amp;_ga=2.51294764.469232365.1667997233-855266531.1667997233#manage-storage-credentials" alt="https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html?&amp;_ga=2.51294764.469232365.1667997233-855266531.1667997233#manage-storage-credentials" target="_blank">https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html#manage-storage-credentials</A></P><P></P><P>then you can use parquet tables.</P><P></P><P>It seems like the way forward is to use managed tables with Unity Catalog, they are gaining some performance improvements over time.</P> Wed, 09 Nov 2022 12:36:24 GMT Pat 2022-11-09T12:36:24Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23138#M15938 <P>We are migrating a job from onprem to databricks. We are trying to optimize the jobs but couldn't use bucketing because by default databricks stores all tables as delta table and it shows error that bucketing is not supported for delta. Is there anyway to do this?</P> Wed, 09 Nov 2022 06:06:47 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23138#M15938 Arun_tsr 2022-11-09T06:06:47Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23140#M15940 <P>Hi @Debayan Mukherjee​&nbsp;We are following similar syntax for creating the bucketed table. But we are getting the following error,</P><P></P><P><I>Operation not allowed: `Bucketing` is not supported for Delta tables</I></P><P></P><P>Databricks by default considering the created tables as delta </P> Wed, 09 Nov 2022 08:40:49 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23140#M15940 Arun_tsr 2022-11-09T08:40:49Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23141#M15941 <P>Hi @Arun Balaji​&nbsp;,</P><P></P><P>bucketing is not supported for the delta tables as you have noticed.</P><P>For the optimization and best practices with delta tables check this:</P><P></P><P><A href="https://docs.databricks.com/optimizations/index.html" target="test_blank">https://docs.databricks.com/optimizations/index.html</A></P><P><A href="https://docs.databricks.com/delta/best-practices.html" target="test_blank">https://docs.databricks.com/delta/best-practices.html</A></P><P></P> Wed, 09 Nov 2022 12:28:37 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23141#M15941 Pat 2022-11-09T12:28:37Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23142#M15942 <P>Is it possible to create a table without making it a delta table​?</P> Wed, 09 Nov 2022 12:33:20 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23142#M15942 Arun_tsr 2022-11-09T12:33:20Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23143#M15943 <P>good question <span class="lia-unicode-emoji" title=":winking_face:">😉</span> I was going to mention this.</P><P></P><P>You can still use external tables. Those tables will be stored outside the main (root) metastore bucket/container.</P><P></P><P><A href="https://docs.databricks.com/data-governance/unity-catalog/create-tables.html" alt="https://docs.databricks.com/data-governance/unity-catalog/create-tables.html" target="_blank">https://docs.databricks.com/data-governance/unity-catalog/create-tables.html</A></P><P></P><P><A href="https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html?&amp;_ga=2.51294764.469232365.1667997233-855266531.1667997233#manage-storage-credentials" alt="https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html?&amp;_ga=2.51294764.469232365.1667997233-855266531.1667997233#manage-storage-credentials" target="_blank">https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html#manage-storage-credentials</A></P><P></P><P>then you can use parquet tables.</P><P></P><P>It seems like the way forward is to use managed tables with Unity Catalog, they are gaining some performance improvements over time.</P> Wed, 09 Nov 2022 12:36:24 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23143#M15943 Pat 2022-11-09T12:36:24Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23144#M15944 <P>you can also check this:</P><P></P><P><A href="https://community.databricks.com/s/question/0D53f00001m1u4qCAA/bucketing-on-delta-tables" target="test_blank">https://community.databricks.com/s/question/0D53f00001m1u4qCAA/bucketing-on-delta-tables</A></P> Wed, 09 Nov 2022 12:48:12 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23144#M15944 Pat 2022-11-09T12:48:12Z https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23139#M15939 <P>Hi @Arun Balaji​&nbsp;, you can go through <A href="https://www.databricks.com/session/bucketing-in-spark-sql-2-3" alt="https://www.databricks.com/session/bucketing-in-spark-sql-2-3" target="_blank">https://www.databricks.com/session/bucketing-in-spark-sql-2-3</A> and also <A href="https://www.databricks.com/session_na20/bucketing-2-0-improve-spark-sql-performance-by-removing-shuffle" alt="https://www.databricks.com/session_na20/bucketing-2-0-improve-spark-sql-performance-by-removing-shuffle" target="_blank">https://www.databricks.com/session_na20/bucketing-2-0-improve-spark-sql-performance-by-removing-shuffle</A> and please let us know if this helps. </P> Wed, 09 Nov 2022 07:33:49 GMT https://community.databricks.com/t5/data-engineering/how-to-do-bucketing-in-databricks/m-p/23139#M15939 Debayan 2022-11-09T07:33:49Z