@ck7007 thanks so much for sharing! That's such a saving, by the way. Congrats.

Out of curiosity, did you consider using Liquid Clustering which was meant to replace partitioning and z-order: https://docs.databricks.com/aws/en/delta/clustering 

I found this part particularly interesting: 

 It provides the flexibility to redefine clustering keys without rewriting existing data, allowing data layout to evolve alongside analytic needs over time

 If you did use it, how did you find that versus Z-Order?

All the best,
BS