topic Re: Clusters are failing to launch in Get Started Discussions

Clusters are failing to launch

Peter_Jones — Mon, 19 Feb 2024 14:02:11 GMT

Hi Guys,
I am a complete newbie to data bricks, we are trying to figure out if our data models and ETL can run on it.

I have got the failure to launch message.

I have read this message as well.
https://community.databricks.com/t5/data-engineering/cluster-sql-warehouse-fails-to-start/m-p/3335#M411

I went into our Azure quota profile and updated the "Total Regional vCPUs" to 40 as requested.

But even waiting 24 hours for the update to go through it is still issuing the same message.

If anyone has the patience to deal with a complete newbie trying to get Data Bricks started on his azure MPN account?

I would be most appreciative. 🙂

Thanks in advance for any assistance you may be able to offer.

Clusters are failing to launch. Cluster launch will be retried.
Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 10, Current Usage: 0, Additional Required: 32, (Minimum) New Limit Required: 32. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"standardEDSv4Family","quotaRequest":{"properties":{"limit":32,"unit":"Count","name":{"value":"standardEDSv4Family"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests Type: CLIENT_ERROR Code: AZURE_QUOTA_EXCEEDED_EXCEPTION Cluster-id (internal): 0219-132302-t9x1s6m1 Additional details: {"azure_error_code":"QuotaExceeded","azure_error_message":"Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 10, Current Usage: 0, Additional Required: 32, (Minimum) New Limit Required: 32. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"standardEDSv4Family","quotaRequest":{"properties":{"limit":32,"unit":"Count","name":{"value":"standardEDSv4Family"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"}

Re: Clusters are failing to launch

Peter_Jones — Mon, 19 Feb 2024 14:55:45 GMT

Hi Kaniz,

ah, I must have allocated some vCPUs that were not "standardEDSv4Family". I tried to allocate 40 of these and it only gave me 30. Presumably because I have 40 of some other family. I am trying to find out what the other ones are I have but can't find how how to differentiate between the two and give back the other 40 vCPUs I asked for.

I will keep looking.

Re: Clusters are failing to launch

Peter_Jones — Mon, 19 Feb 2024 15:26:35 GMT

Ok Kaniz, I found how to determine how many resources were allocated to my account. You can see EDSV4 is allocated 16 with a limit 40 now.

But now I am at least getting a different message. I do not see how to reduce the total regional vCPUs. I think that is the problem.

Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 70, Current Usage: 40, Additional Required: 32, (Minimum) New Limit Required: 72. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"cores","quotaRequest":{"properties":{"limit":72,"unit":"Count","name":{"value":"cores"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests Type: CLIENT_ERROR Code: AZURE_QUOTA_EXCEEDED_EXCEPTION Cluster-id (internal): 0219-151611-dh2zm02d Additional details: {"azure_error_code":"QuotaExceeded","azure_error_message":"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 70, Current Usage: 40, Additional Required: 32, (Minimum) New Limit Required: 72. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"cores","quotaRequest":{"properties":{"limit":72,"unit":"Count","name":{"value":"cores"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests"}

Re: Clusters are failing to launch

Peter_Jones — Mon, 19 Feb 2024 15:35:20 GMT

Ok, I am now at 32 of 40 EDSV4 vCPUs. I will see if I can get databricks started now.

Re: Clusters are failing to launch

Peter_Jones — Mon, 19 Feb 2024 16:51:04 GMT

Hi Kaniz,

I got another message about needing more public IP addresses and I increased them to 20.

Now I have another message saying it needs 104 cores.

This seems unreasonable for a "starter data warehouse" which is what I am trying to start. So I think it's some sort of error, not that I would know. You can check the image below that 48 cores out of 72 quota are allocated.

I thought it only needed 32. So I am guessing I am doing something wrong and I don't know what.

Any guidance you, or others, can offer would be greatly appreciated.

(A quick update! It now says it is running!!! HOORAY!!)

Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 82, Current Usage: 72, Additional Required: 32, (Minimum) New Limit Required: 104. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"cores","quotaRequest":{"properties":{"limit":104,"unit":"Count","name":{"value":"cores"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests Type: CLIENT_ERROR Code: AZURE_QUOTA_EXCEEDED_EXCEPTION Cluster-id (internal): 0219-163655-6ul1fftb Additional details: {"azure_error_code":"QuotaExceeded","azure_error_message":"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 82, Current Usage: 72, Additional Required: 32, (Minimum) New Limit Required: 104. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/{"subscriptionId":"3e893a92-ca05-4c4c-9344-81886f6c37ca","command":"openQuotaApprovalBlade","quotas":[{"location":"westeurope","providerId":"Microsoft.Compute","resourceName":"cores","quotaRequest":{"properties":{"limit":104,"unit":"Count","name":{"value":"cores"}}}}]} by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests"}

Re: Clusters are failing to launch

Peter_Jones — Wed, 21 Feb 2024 12:35:20 GMT

Hi Kaniz,

I have been able to get the database started and to create some delta tables.

So all good.

Thank you very much for your help.

We will test our ETL on it this weekend and see what bugs/problems we find.

We are sure to find a few. And then we can figure out how much effort it is to apply the needed updates.

Best Regards

Peter