Hi @yliu , using multithreading can indeed help with the performance of Z-ordering optimization on multiple Delta tables. It is mentioned that the OPTIMIZE command has been improved to commit batches as soon as possible, instead of at the end, and the default number of threads OPTIMIZE runs in parallel has been reduced, which is a strict performance increase for large tables.
However, it's important to note that the effectiveness of multithreading will depend on the specific characteristics of your tables and your computing environment. For example, if your tables are very large and have many columns, using multithreading might speed up the process significantly. On the other hand, if your tables are smaller and have fewer columns, the improvement might be less noticeable.Furthermore, the OPTIMIZE operation now uses Hilbert space-filling curves by default, which provides better clustering characteristics than Z-order in higher dimensions. This approach can speed up read queries by skipping more data than Z-order.
So, based on your testing in a dev environment and the provided information, it does make sense to use multithreading to speed up the Z-ordering optimization process.
However, as the tables in dev are not updated very frequently, it would be advisable to continue monitoring and testing this approach in your production environment to ensure it continues to provide the desired performance improvements.
