Clusters are failing to launch

[Peter_Jones]

Hi Guys,
I am a complete newbie to data bricks, we are trying to figure out if our data models and ETL can run on it.

I have got the failure to launch message. 

I have read this message as well.
https://community.databricks.com/t5/data-engineering/cluster-sql-warehouse-fails-to-start/m-p/3335#M...

I went into our Azure quota profile and updated the "Total Regional vCPUs" to 40 as requested.

But even waiting 24 hours for the update to go through it is still issuing the same message.

If anyone has the patience to deal with a complete newbie trying to get Data Bricks started on his azure MPN account?

I would be most appreciative. 🙂 

Thanks in advance for any assistance you may be able to offer.

Clusters are failing to launch. Cluster launch will be retried.
Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 10, Current Usage: 0, Additional Required: 32, (Minimum) New Limit Required: 32. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"standardEDSv4Family","quotaRequest":{"properties":{"limit":32,"unit":"Count","name":{"value":"standardEDSv4Family"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests Type: CLIENT_ERROR Code: AZURE_QUOTA_EXCEEDED_EXCEPTION Cluster-id (internal): 0219-132302-t9x1s6m1 Additional details: {"azure_error_code":"QuotaExceeded","azure_error_message":"Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 10, Current Usage: 0, Additional Required: 32, (Minimum) New Limit Required: 32. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"standardEDSv4Family","quotaRequest":{"properties":{"limit":32,"unit":"Count","name":{"value":"standardEDSv4Family"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"}

[Kaniz_Fatma · Accepted solution]

Hi @Peter_Jones, Let’s tackle this cluster launch issue step by step.

1. Quota Exceeded Error: The error message indicates that your cluster launch failed due to exceeding the approved quota for standardEDSv4Family Cores in the westeurope location.

2. Here’s what you can do:

   • Check Current Quota: Verify the current quota limit for standardEDSv4Family Cores. You mentioned that you updated it to 40, but the error still persists. Make sure the update has propagated across your Azure subscription.

   • Request Quota Increase: If the current limit is insufficient, you’ll need to request a quota increase. Follow these steps:

     • Visit the Azure portal.
     • Navigate to Usage + quotas.
     • Select your subscription.
     • Look for standardEDSv4Family Cores in the list and click on it.
     • Click Request increase and provide the necessary details.
     • Specify the new limit (at least 32 in your case) and submit the request.

   • Details Section: The error message provides a link to submit the quota increase request. You can find it here.

   • Learn More: For a deeper understanding of quota limits, refer to the official documentation.
     
     

   • If you have any more questions or need further assistance, feel free to ask. 🚀

[Peter_Jones]

Hi Kaniz,

ah, I must have allocated some vCPUs that were not "standardEDSv4Family". I tried to allocate 40 of these and it only gave me 30. Presumably because I have 40 of some other family. I am trying to find out what the other ones are I have but can't find how how to differentiate between the two and give back the other 40 vCPUs I asked for.

I will keep looking.

[Peter_Jones]

Ok Kaniz, I found how to determine how many resources were allocated to my account. You can see EDSV4 is allocated 16 with a limit 40 now.

But now I am at least getting a different message. I do not see how to reduce the total regional vCPUs. I think that is the problem.

Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 70, Current Usage: 40, Additional Required: 32, (Minimum) New Limit Required: 72. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"su... by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests Type: CLIENT_ERROR Code: AZURE_QUOTA_EXCEEDED_EXCEPTION Cluster-id (internal): 0219-151611-dh2zm02d Additional details: {"azure_error_code":"QuotaExceeded","azure_error_message":"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 70, Current Usage: 40, Additional Required: 32, (Minimum) New Limit Required: 72. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"su... by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests"}

[Peter_Jones]

Ok, I am now at 32 of 40 EDSV4 vCPUs.  I will see if I can get databricks started now.

[Peter_Jones]

Hi Kaniz,

I got another message about needing more public IP addresses and I increased them to 20.

Now I have another message saying it needs 104 cores.

This seems unreasonable for a "starter data warehouse" which is what I am trying to start. So I think it's some sort of error, not that I would know. You can check the image below that 48 cores out of 72 quota are allocated.

I thought it only needed 32. So I am guessing I am doing something wrong and I don't know what.

Any guidance you, or others, can offer would be greatly appreciated.

(A quick update! It now says it is running!!! HOORAY!!) 

Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 82, Current Usage: 72, Additional Required: 32, (Minimum) New Limit Required: 104. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"cores","quotaRequest":{"properties":{"limit":104,"unit":"Count","name":{"value":"cores"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests Type: CLIENT_ERROR Code: AZURE_QUOTA_EXCEEDED_EXCEPTION Cluster-id (internal): 0219-163655-6ul1fftb Additional details: {"azure_error_code":"QuotaExceeded","azure_error_message":"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 82, Current Usage: 72, Additional Required: 32, (Minimum) New Limit Required: 104. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"cores","quotaRequest":{"properties":{"limit":104,"unit":"Count","name":{"value":"cores"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests"}

[Peter_Jones]

Hi Kaniz,

I have been able to get the database started and to create some delta tables.

So all good.

Thank you very much for your help.

We will test our ETL on it this weekend and see what bugs/problems we find.

We are sure to find a few. And then we can figure out how much effort it is to apply the needed updates.

Best Regards 

Peter