topic Re: Job queue for pool limit in Data Engineering

Job queue for pool limit

andyh — Thu, 09 Nov 2023 10:35:30 GMT

I have a cluster pool with a max capacity limit, to make sure we're not burning too extra silicon. We use this for some of our less critical workflow/jobs. They still spend a lot of time idle, but sometimes hit this max capacity limit. Is there a way to get a job to wait for an available pool instance, rather than automatically failing with an

instance_pool_error_code:INSTANCE_POOL_MAX_CAPACITY_FAILURE

Re: Job queue for pool limit

karthik_p — Fri, 10 Nov 2023 18:33:32 GMT

@andyh did you get a chance to check queue in jobs, that may help, will update if we have any other options

Re: Job queue for pool limit

SSundaram — Fri, 10 Nov 2023 20:18:41 GMT

Try increasing your max capacity limit and might want to bring down the min number of nodes the job uses.

At the job level try configuring retry and time interval between retries.