@tanin​ 

• When you do repartition(10000), it will create 10000 tasks which will take time to launch especially if the number of cores available in your clusters is less. e.g. If you have 100 cores available in the cluster, it would have to run 100 times (100 * 100 = 10000) to complete that many tasks
• Use the repartition number such that data block size on each task is distributed evenly and the block size is ~128MB
• Avoid using collect as after doing repartition (which will distribute data amongst executors), collect will try to bring it back to a single driver which is a heavy shuffle operation and will use lot of IO

If this helps, please upvote the answer