Should I use Z Ordering on my Delta table every time I run Optimize?

[User16826992666]

Wondering if it always makes sense or if there are some situations where you might only want to run optimize

[Srikanth_Gupta_ · Accepted solution]

Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operations

ZORDER BY ->

Colocate column information in the same set of files. Co-locality is used by Delta Lake data-skipping algorithms to dramatically reduce the amount of data that needs to be read. You can specify multiple columns for ZORDER BY  as a comma-separated list. However, the effectiveness of the locality drops with each additional column. Z-Ordering is not idempotent but aims to be an incremental operation.