topic Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete? in Data Engineering

Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Matt101122 — Wed, 12 Oct 2022 14:39:02 GMT

We are having an issue with calls to /api/2.1/jobs/create taking between 30 and 60 seconds to complete. Calls to /api/2.1/jobs/list complete as expected in about 6 seconds so the issue seems to be only with the job create API. This seems to only be happening on our premium workspaces. Calls from our standard workspace complete in 6 seconds. This delay is causing our job deployment processing to take a long time to complete as we have many jobs that get updated during deployment.

Note: workspaces are in Azure EastUS2

Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Matt101122 — Tue, 18 Oct 2022 19:24:51 GMT

@Debayan Mukherjee I understand that the create endpoint would take longer but there is a significant difference between our standard workspace and our premium workspace. The standard is much faster whereas premium is slower... is there anything else network or route related that could impact this?

Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Matt101122 — Thu, 20 Oct 2022 12:58:34 GMT

@Debayan Mukherjee the issue has been resolved! Basically all calls to a domain name were slow.

We determined that traffic couldn't get to our primary azure dns so it was using the corporate dns on-prem. Now that a networking issue has been resolved with traffic getting to azure dns call are running as expected. A call to update a job now completes in less than 1 second. 👍

Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Anonymous — Sat, 19 Nov 2022 15:01:29 GMT

Hi @Matthew Dalesio

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Debayan — Tue, 18 Oct 2022 16:03:56 GMT

Hi @Matthew Dalesio , /api/2.1/jobs/list only lists the existing jobs which does not take much time, where as /api/2.1/jobs/create is meant to be taking time as it creates a job cluster in the background which involves few other internal workflow. Please let us know if this answers your question.