When deploying and running several jobs at the same time we get the error: REQUEST_LIMIT_EXCEEDED: Your request was rejected since cluster creation, start and upsize requests within your organization have exceeded the rate limit of 100 nodes per minute. Please retry your request later, or choose a larger node type instead.

[Adam_Pavlacka]

There is a rate limit of 100 notes per minute. To ensure you do not exceed this limit, you should make adjustments to the deployment and execution of your ML jobs.

Distribute recurring workflows evenly over the planned time period

To ensure compliance with the limit of 100 nodes per minute, it is recommended to slow down the rate of cluster creation, initiation, and upscaling requests. You can achieve this by implementing a rate-limiting mechanism in your code. For instance, you can introduce delays between each request or restrict the number of requests per minute.

Select a larger node type

As indicated by the error message, an alternative approach is to choose a larger node type instead of creating multiple smaller clusters. By opting for a larger node type, you can stay within the rate limit while still providing sufficient resources for your ML jobs.

Optimize your code

Evaluate your code to identify any areas that may be responsible for unnecessary cluster creation or inefficient resource utilization. By optimizing your code, you can potentially reduce the number of requests for cluster creation, initiation, and upscaling.