10-05-2021 09:52 AM
When running a jar-based job, I've noticed that the 1st run always takes the extra time to complete the job and consecutive runs take less time to finish the job. This behavior is reproducible on an interactive cluster. What's causing this? Is this expected?
10-06-2021 04:56 AM
Hi @Brad Barker , In the 1st run of the jar-based job when the 1st job/task starts executor fetches the jar/library from driver to executor. Which will take some time but in the 2nd run or any consecutive runs, there will be no fetch operation so all consecutive runs will take less time to finish the job compared to the 1st run. This is expected behavior.
Overall there might be other reasons for job slowness like slow network, large library, more input data. But in an ideal scenario, the 1st run (after the cluster is started) ships the libraries to executors with the first task, and we may see a couple of seconds/minutes of slowness (based on the number & size of the libraries).
10-05-2021 10:45 PM
Hi @ Brad! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will follow up with my team and get back to you soon. Thanks.
10-06-2021 12:43 AM
Hi @Brad Barker
Does the jar job has dependencies ?. Can you elaborate what does this jar job does ?.
10-06-2021 01:35 AM
maybe the loading of the jar takes some time? When you rerun it, it might already be present.
Just thinking out loud here.
10-06-2021 04:56 AM
Hi @Brad Barker , In the 1st run of the jar-based job when the 1st job/task starts executor fetches the jar/library from driver to executor. Which will take some time but in the 2nd run or any consecutive runs, there will be no fetch operation so all consecutive runs will take less time to finish the job compared to the 1st run. This is expected behavior.
Overall there might be other reasons for job slowness like slow network, large library, more input data. But in an ideal scenario, the 1st run (after the cluster is started) ships the libraries to executors with the first task, and we may see a couple of seconds/minutes of slowness (based on the number & size of the libraries).
10-06-2021 05:58 AM
@Sandeep Katta , this is a fat jar that does read-transform-write. @DD Sharma response matches
@Werner Stinckens & I intuition that there was efficiency on the second job due to jar already being loaded. I would not have noticed this had job run time not been small. There is follow on opportunity here on how to make library installs faster, but this answers my question.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.