https://community.databricks.com/t5/data-engineering/how-to-set-file-size-for-merge/m-p/91817#M38272 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/100643">@MikeGo</a>&nbsp;,</P><P>There are a couple of considerations here, the main being your runtime version and also whether you are using unit catalog.</P><P>Check this document:<BR /><A href="https://docs.databricks.com/en/delta/tune-file-size.html" target="_blank">https://docs.databricks.com/en/delta/tune-file-size.html</A><BR /><BR /><BR /></P> Thu, 26 Sep 2024 07:09:24 GMT filipniziol 2024-09-26T07:09:24Z https://community.databricks.com/t5/data-engineering/how-to-set-file-size-for-merge/m-p/91811#M38269 <P>Hi team,</P><P>I use MERGE to merge source to target table. Source is incremental reading with checkpoint on delta table. Target is delta table without any partition. If the table is empty, with&nbsp;<SPAN>spark.databricks.delta.optimizeWrite.enabled it can create files in around 128M per file. However next time run it shows meany small files. Is there a way to control the file size after MERGE?</SPAN></P><P>Thanks</P> Thu, 26 Sep 2024 06:33:08 GMT https://community.databricks.com/t5/data-engineering/how-to-set-file-size-for-merge/m-p/91811#M38269 MikeGo 2024-09-26T06:33:08Z https://community.databricks.com/t5/data-engineering/how-to-set-file-size-for-merge/m-p/91817#M38272 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/100643">@MikeGo</a>&nbsp;,</P><P>There are a couple of considerations here, the main being your runtime version and also whether you are using unit catalog.</P><P>Check this document:<BR /><A href="https://docs.databricks.com/en/delta/tune-file-size.html" target="_blank">https://docs.databricks.com/en/delta/tune-file-size.html</A><BR /><BR /><BR /></P> Thu, 26 Sep 2024 07:09:24 GMT https://community.databricks.com/t5/data-engineering/how-to-set-file-size-for-merge/m-p/91817#M38272 filipniziol 2024-09-26T07:09:24Z