Submitting multiple parallel jobs to the same job cluster causes Azure vCPU quota manager to count the clusters vCPUs on each invocation

[labromb]

I have an ADF pipeline which invokes a Databricks job six times in parallel. My assumption is all jobs get routed to the same job cluster which then deals with all the invocations in parallel. This was working fine when I had five sources, when I added the six I got this error

Error code: OperationNotAllowed, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 350, Current Usage: 347, Additional Required: 4, (Minimum) New Limit Required: 351. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2299d49066-62e5-4704-acd8-6b6ef4278d07%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22westeurope%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22cores%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:351,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22cores%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests (Azure request Id: 5306171b-4929-43a9-a711-aaba42a4e5f3)

My cluster has a driver and eight workers (all 4 core). If i multiply the number of cluster cores by the number of invocations and then double to account for hyper threading I can see that the 6th invocation would trip the 350 quota we currently have in play.

This doesn't seem right to me, as I am only spinning up one job cluster. Is the quota manager getting confused?

Any comments greatly appreciated.

[-werners-]

each job has its cluster, that is how it is designed.

So either request a quotum increase or run the jobs serially or use notebook workflows (run multiple notebooks in parallel on a single cluster)/cluster pools

[Hubert-Dudek]

Yes exactly. If you want to use just one cluster, please set it sequentially in ADF.

[labromb]

Many thanks both for confirming.

[jose_gonzalez]

Hi @Brian Labrom​,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.