Can't use job cluster for scheduled jobs ADD_NODES_FAILED : Failed to add 9 containers to the cluster. Will attempt retry: false. Reason: Azure Quota Exceeded Exception

[B_J_Innov]

Hi everyone,

I've been using my all purpose cluster for scheduled jobs and I've been told that it's a suboptimal thing to do and that using a job cluster for the scheduled jobs cuts costs by half.

Unfortunately, when I tried to switch clusters on my existing job, an unexpected error appeared : Unexpected failure while waiting for the cluster to be ready : is in unexpected state Terminated: AZURE_QUOTA_EXCEEDED_EXCEPTION(CLIENT_ERROR):

The operation couldn't be completed. Databricks quota estimation shows that you don't have sufficient CPU core for instance type Standard_DS3_v2 to launch the cluster. Consider increasing quota for the instance type, decreasing quota usage, trying other instance types and trying again later,databricks_error_message:The operation couldn't be completed. Databricks quota estimation shows that you don't have sufficient CPU core for instance type Standard_DS3_v2 to launch the cluster. Consider increasing quota for the instance type, decreasing quota usage, trying other instance types and trying again later.

I don't know why this happens, I'm supposed to have a job cluster and I know that nobody uses it.

It seems like nobody had this error in this forum that's why I post this issue, thanks in advance for your help !

[karthik_p · Accepted solution]

@Bassem Jaber​ quota limit is related to your azure cloud, you can go to azure quota page and select particular instance and increase limit and you should be good. if you are in pay as go subscription from cloud then you will have limitations on quota increase also

[B_J_Innov]

@karthik p​ Thank you for your answer !

I'm in a Pay-as-you-go subscription and I have several quotas in several regions.

There is one that is near the limit and I guess it is the one that I use by default, but there is another that is almost not used. Is it possible to use this ressource in a specific region for the job cluster ?

[karthik_p]

@Bassem Jaber​ try to pick different instance type, that should resolve your issue. if you are seeing resource with instance type then that can be used

[B_J_Innov]

@karthik p​ When I try to create a new job cluster, unfortunately, it doesn't allow me to select the region

[karthik_p]

@Bassem Jaber​ you can not select from different region ( as region is tied up with your workspace), you need to select different instance type other than Standard_DS3_v2

[Anonymous]

Hi @Bassem Jaber​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

[B_J_Innov]

Hi @Vidula Khanna​

I understood the root of the issue but I'd like to be helped in order to understand if adding quota and running jobs on a job cluster is cheaper than running jobs on an all purpose cluster with my actual setup.

Is it the right place to do it or should I ask somewhere else ?

[karthik_p]

@Bassem Jaber​ yes, for sure all purpose clusters always 2 X time's pricer than job cluster . First create your job in all purpose cluster by tuning your cluster size. once you are comfortable with performance, use same config for creating job cluster and schedule your jobs.

as job cluster will get trigger based on schedule or type of config you choose and will stop once after completion of job @Vidula Khanna​ 

[B_J_Innov]

@karthik p​ I can create compute for All-purpose Compute but not Job Compute, is it because I don't have the needed rights ?

[karthik_p]

@Bassem Jaber​ are you seeing any error, when you create job you should be able to see cluster option --> in that select new job cluster. if you are not owner of that job you can create job cluster

[B_J_Innov]

@karthik p​ I don't see any error when creating je job cluster and the job, I just can't use a quota in another instance I think, it always throws the same error

[karthik_p]

@Bassem Jaber​ If you are seeing same error then you need to increase quota, for that your azure plan should be changed from pay as you go to other plan. as pay-as-go azure model has limitations on quota increase