cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Should I use Z Ordering on my Delta table every time I run Optimize?

User16826992666
Valued Contributor

Wondering if it always makes sense or if there are some situations where you might only want to run optimize

1 ACCEPTED SOLUTION

Accepted Solutions

Srikanth_Gupta_
Valued Contributor

Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operations

ZORDER BY ->

Colocate column information in the same set of files. Co-locality is used by Delta Lake data-skipping algorithms to dramatically reduce the amount of data that needs to be read. You can specify multiple columns for ZORDER BY  as a comma-separated list. However, the effectiveness of the locality drops with each additional column. Z-Ordering is not idempotent but aims to be an incremental operation. 

View solution in original post

1 REPLY 1

Srikanth_Gupta_
Valued Contributor

Its good idea to optimize at end of each batch job to avoid any small files situation, Z order is optional and can be applied on few non partition columns which are used frequently in read operations

ZORDER BY ->

Colocate column information in the same set of files. Co-locality is used by Delta Lake data-skipping algorithms to dramatically reduce the amount of data that needs to be read. You can specify multiple columns for ZORDER BY  as a comma-separated list. However, the effectiveness of the locality drops with each additional column. Z-Ordering is not idempotent but aims to be an incremental operation. 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.