How to set file size for MERGE

[Brad]

Hi team,

I use MERGE to merge source to target table. Source is incremental reading with checkpoint on delta table. Target is delta table without any partition. If the table is empty, with spark.databricks.delta.optimizeWrite.enabled it can create files in around 128M per file. However next time run it shows meany small files. Is there a way to control the file size after MERGE?

Thanks

[filipniziol]

Hi @Brad ,

There are a couple of considerations here, the main being your runtime version and also whether you are using unit catalog.

Check this document:
https://docs.databricks.com/en/delta/tune-file-size.html