How to set file size for MERGE

MikeGo
Valued Contributor

Hi team,

I use MERGE to merge source to target table. Source is incremental reading with checkpoint on delta table. Target is delta table without any partition. If the table is empty, with spark.databricks.delta.optimizeWrite.enabled it can create files in around 128M per file. However next time run it shows meany small files. Is there a way to control the file size after MERGE?

Thanks

filipniziol
Esteemed Contributor

Hi @MikeGo ,

There are a couple of considerations here, the main being your runtime version and also whether you are using unit catalog.

Check this document:
https://docs.databricks.com/en/delta/tune-file-size.html