Z-ordering optimization with multithreading

[yliu]

Hi, 

I am wondering if multithreading will help with the performance for z-ordering optimization on multiple delta tables.

We are periodically doing optimization on thousands of tables and it easily takes a few days to finish the job. So we are looking for a way to optimize a number of tables in parallel. Will using multithreading make sense here speed up the process? We did a few rounds of testing in dev environment and it seems the optimization with multithreading does a better job. But we couldn't be sure as the tables in dev are not updated very frequently so sometimes the optimization is not actually writing anything. 

Thank you in advance for your help!

[yliu]

Thank you for the detailed explanation and quick response! I will proceed with multithreading then and keep monitoring it. Thanks a lot :))

[yliu]

Hi Kaniz, 

A follow up question on this topic. How does python multithreading works with Spark? We are trying to understand how to decide the number of threads to maximum performance. And which one is better, multithreading or multiprocessing? Some of our tests show that, multiprocessing are faster. Could you shed some lights?

Thank you!