hello all

while reading content that provides guidance on delta lake file size, i realized tuneFileSizesForRewrites behind the scenes targets for 256 MB file size.

and optimize.maxFileSize will target for 1 GB file ((reference : https://docs.databricks.com/aws/en/sql/language-manual/delta-optimize)).

In our environment, we have tried using tuneFileSizesForRewrites and we saw around 86 MB size files for a table but optimize did not make them bigger, not even 126 MB, forget about 1 GB.

also, I read from 'data lake - definitive guide' book that prior to DBR 8.4 - tuneFileSizesForRewrites was based on the number of merge operations happened during the last 10 operations, as on, the overall idea was to create smaller files for merge performance improvement. We realized this in some form while testing to decide between tuneFileSizesForRewrites & setting the targetFileSize manually - if we specify targetFileSize = 256MB unlike our current targetFileSize = 33 MB, the MERGE performance (bronze > silver layer) was slower. Now, few questions -

• in our environment, optimize did not make 1 gb files, is that the case with you all
• if optimize really makes bigger files of around 1 gb, will merge performance be not be a problem then?
• in the raw (plain parquet/source mimic) > bronze layer load which is append-only, can we specify a very big targetFileSize? In the 'data lake - definitive guide' book, the author makes the claim - 'High-volume append-only tables in the bronze layer generally function better with larger file sizes, as the larger sizes maximize throughput per operation with little regard to anything else.'