topic Will Unsetting delta.targetFileSize During Data Load Cause Any Issues? in Data Engineering

Will Unsetting delta.targetFileSize During Data Load Cause Any Issues?

pooja_bhumandla — Wed, 30 Jul 2025 10:33:00 GMT

Hi,
If I unset the Spark config delta.targetFileSize (e.g., using alter) while a data load is in progress (batch or streaming), will it cause any issues?

Will the load fail or behave inconsistently due to the config being changed mid-process?

Thanks!

Re: Will Unsetting delta.targetFileSize During Data Load Cause Any Issues?

mani_22 — Wed, 30 Jul 2025 12:40:49 GMT

@pooja_bhumandla delta.targetFileSize controls the file size for operations including optimise or Z-order, auto compaction, and optimised writes.

Unsetting this config during data load will not cause any failures or behave inconsistently. After unsetting, the above operations will use the default value as the target file sizes.

Re: Will Unsetting delta.targetFileSize During Data Load Cause Any Issues?

Brahmareddy — Wed, 30 Jul 2025 13:28:43 GMT

Hi pooja_bhumandla,

How are you doing today? In general, changing the delta.targetFileSize config while a batch or streaming load is in progress won’t crash your job, but it may lead to inconsistent behavior during that specific run. Spark jobs usually pick up the config at the start of execution, so if you unset or change it mid-load, only the subsequent operations or triggers might reflect the new setting. This could result in a mix of file sizes being written to Delta, which might not be ideal if you’re aiming for consistent file sizing. So, while it's not likely to fail your job outright, it’s best to change such configurations between loads or restarts to avoid unexpected results. Hope that helps!

Regards,

Brahma