How to set file size for MERGE
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-25-2024 11:33 PM
Hi team,
I use MERGE to merge source to target table. Source is incremental reading with checkpoint on delta table. Target is delta table without any partition. If the table is empty, with spark.databricks.delta.optimizeWrite.enabled it can create files in around 128M per file. However next time run it shows meany small files. Is there a way to control the file size after MERGE?
Thanks
- Labels:
-
Delta Lake
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2024 12:09 AM
Hi @Brad ,
There are a couple of considerations here, the main being your runtime version and also whether you are using unit catalog.
Check this document:
https://docs.databricks.com/en/delta/tune-file-size.html
![](/skins/images/8C2A30E5B696B676846234E4B14F2C7B/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/8C2A30E5B696B676846234E4B14F2C7B/responsive_peak/images/icon_anonymous_message.png)