Z-ordering optimization with multithreading
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-18-2023 04:35 AM - edited 09-18-2023 04:36 AM
Hi,
I am wondering if multithreading will help with the performance for z-ordering optimization on multiple delta tables.
We are periodically doing optimization on thousands of tables and it easily takes a few days to finish the job. So we are looking for a way to optimize a number of tables in parallel. Will using multithreading make sense here speed up the process? We did a few rounds of testing in dev environment and it seems the optimization with multithreading does a better job. But we couldn't be sure as the tables in dev are not updated very frequently so sometimes the optimization is not actually writing anything.
Thank you in advance for your help!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-19-2023 01:39 AM
Thank you for the detailed explanation and quick response! I will proceed with multithreading then and keep monitoring it. Thanks a lot :))
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-05-2023 12:41 AM - edited 10-05-2023 12:50 AM
Hi Kaniz,
A follow up question on this topic. How does python multithreading works with Spark? We are trying to understand how to decide the number of threads to maximum performance. And which one is better, multithreading or multiprocessing? Some of our tests show that, multiprocessing are faster. Could you shed some lights?
Thank you!