User16826994223
Databricks Employee
Databricks Employee

A better way I can think of is-

Enable auto optimize (it will automatically create a file of 128 mb)

Enable Auto compact.

delta.autoOptimize.optimizeWrite = true
 delta.autoOptimize.autoCompact = true 

​complete guide-

https://docs.microsoft.com/en-us/azure/databricks/delta/optimizations/auto-optimize