I have an ADF pipeline which invokes a Databricks job six times in parallel. My assumption is all jobs get routed to the same job cluster which then deals with all the invocations in parallel. This was working fine when I had five sources, when I added the six I got this error
Error code: OperationNotAllowed, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 350, Current Usage: 347, Additional Required: 4, (Minimum) New Limit Required: 351. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2299d49066-62e5-4704-acd8-6b6ef4278d07%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22westeurope%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22cores%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:351,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22cores%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests (Azure request Id: 5306171b-4929-43a9-a711-aaba42a4e5f3)
My cluster has a driver and eight workers (all 4 core). If i multiply the number of cluster cores by the number of invocations and then double to account for hyper threading I can see that the 6th invocation would trip the 350 quota we currently have in play.
This doesn't seem right to me, as I am only spinning up one job cluster. Is the quota manager getting confused?
Any comments greatly appreciated.
