topic Re-optimize in delta not splitting large files to smaller files. in Data Engineering

Re-optimize in delta not splitting large files to smaller files.

sajith_appukutt — Wed, 09 Jun 2021 07:07:20 GMT

I am trying to re-optimize the a delta table with a max file size of 32 MB. But after changing spark.databricks.delta.optimize.maxFileSize and trying to optimize a partition, it doesn't split larger files to smaller ones. How can i get it to work.

Re: Re-optimize in delta not splitting large files to smaller files.

sajith_appukutt — Mon, 21 Jun 2021 22:21:11 GMT

spark.databricks.delta.optimize.maxFileSize controls the target size to binpack files when you run OPTIMIZE command. But it will not split larger files to smaller ones today. File splitting happens when ZORDER is ran however.