Should I use Z Ordering on my Delta table every time I run Optimize?

User16826992666 — Thu, 17 Jun 2021 03:52:14 GMT

Wondering if it always makes sense or if there are some situations where you might only want to run optimize

Re: Should I use Z Ordering on my Delta table every time I run Optimize?

Srikanth_Gupta_ — Thu, 17 Jun 2021 19:47:45 GMT

Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operations

ZORDER BY ->

Colocate column information in the same set of files. Co-locality is used by Delta Lake data-skipping algorithms to dramatically reduce the amount of data that needs to be read. You can specify multiple columns for ZORDER BY as a comma-separated list. However, the effectiveness of the locality drops with each additional column. Z-Ordering is not idempotent but aims to be an incremental operation.

topic Should I use Z Ordering on my Delta table every time I run Optimize? in Data Engineering

Should I use Z Ordering on my Delta table every time I run Optimize?

Re: Should I use Z Ordering on my Delta table every time I run Optimize?