cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Can't use job cluster for scheduled jobs ADD_NODES_FAILED : Failed to add 9 containers to the cluster. Will attempt retry: false. Reason: Azure Quota Exceeded Exception

B_J_Innov
New Contributor III

Hi everyone,

I've been using my all purpose cluster for scheduled jobs and I've been told that it's a suboptimal thing to do and that using a job cluster for the scheduled jobs cuts costs by half.

Unfortunately, when I tried to switch clusters on my existing job, an unexpected error appeared : Unexpected failure while waiting for the cluster to be ready : is in unexpected state Terminated: AZURE_QUOTA_EXCEEDED_EXCEPTION(CLIENT_ERROR):

The operation couldn't be completed. Databricks quota estimation shows that you don't have sufficient CPU core for instance type Standard_DS3_v2 to launch the cluster. Consider increasing quota for the instance type, decreasing quota usage, trying other instance types and trying again later,databricks_error_message:The operation couldn't be completed. Databricks quota estimation shows that you don't have sufficient CPU core for instance type Standard_DS3_v2 to launch the cluster. Consider increasing quota for the instance type, decreasing quota usage, trying other instance types and trying again later.

I don't know why this happens, I'm supposed to have a job cluster and I know that nobody uses it.

It seems like nobody had this error in this forum that's why I post this issue, thanks in advance for your help !

1 ACCEPTED SOLUTION

Accepted Solutions

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ quota limit is related to your azure cloud, you can go to azure quota page and select particular instance and increase limit and you should be good. if you are in pay as go subscription from cloud then you will have limitations on quota increase also

View solution in original post

12 REPLIES 12

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ quota limit is related to your azure cloud, you can go to azure quota page and select particular instance and increase limit and you should be good. if you are in pay as go subscription from cloud then you will have limitations on quota increase also

B_J_Innov
New Contributor III

@karthik pโ€‹ Thank you for your answer !

I'm in a Pay-as-you-go subscription and I have several quotas in several regions.

There is one that is near the limit and I guess it is the one that I use by default, but there is another that is almost not used. Is it possible to use this ressource in a specific region for the job cluster ?

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ try to pick different instance type, that should resolve your issue. if you are seeing resource with instance type then that can be used

B_J_Innov
New Contributor III

@karthik pโ€‹ When I try to create a new job cluster, unfortunately, it doesn't allow me to select the region

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ you can not select from different region ( as region is tied up with your workspace), you need to select different instance type other than Standard_DS3_v2

Anonymous
Not applicable

Hi @Bassem Jaberโ€‹ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

B_J_Innov
New Contributor III

Hi @Vidula Khannaโ€‹

I understood the root of the issue but I'd like to be helped in order to understand if adding quota and running jobs on a job cluster is cheaper than running jobs on an all purpose cluster with my actual setup.

Is it the right place to do it or should I ask somewhere else ?

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ yes, for sure all purpose clusters always 2 X time's pricer than job cluster . First create your job in all purpose cluster by tuning your cluster size. once you are comfortable with performance, use same config for creating job cluster and schedule your jobs.

as job cluster will get trigger based on schedule or type of config you choose and will stop once after completion of job @Vidula Khannaโ€‹ 

B_J_Innov
New Contributor III

@karthik pโ€‹ I can create compute for All-purpose Compute but not Job Compute, is it because I don't have the needed rights ?

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ are you seeing any error, when you create job you should be able to see cluster option --> in that select new job cluster. if you are not owner of that job you can create job cluster

B_J_Innov
New Contributor III

@karthik pโ€‹ I don't see any error when creating je job cluster and the job, I just can't use a quota in another instance I think, it always throws the same error

karthik_p
Esteemed Contributor

@Bassem Jaberโ€‹ If you are seeing same error then you need to increase quota, for that your azure plan should be changed from pay as you go to other plan. as pay-as-go azure model has limitations on quota increase

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group