Databricks pool - 2 instances are in running state without any job running in the system

[eager_to_learn]

We are using Azure Databricks pools, configured 16 max instances. Out of 16, 2 instances are in running state without any job in running condition, how & where can i check the usage of the instances ?

p.s. SQL pool is also not running, so no chances of instances being consumed there.

[Prabakar]

hi @Arun Manocha​ It could be possible that you have min idle set to 2. This is the minimum number of instances the pool will keep idle. These instances do not terminate, regardless of the setting specified in Idle Instance Auto Termination.

[eager_to_learn]

Hi @Prabakar Ammeappin​ 

No, we have kept the idle as 0 and moreover its keeping 2 jobs always running which is causing capacity constraint for us and its getting difficult to find out where its running.

[Prabakar]

@Arun Manocha​, You can check it in the Azure portal. I believe the running instances should have the pool name tagged. So you can do a backtrace and check the cluster tag for that instance. Then take the cluster-ID to the Databricks clusters page and use the cluster-ID in the cluster URL to get the running cluster. Once you get the group, you can find what is using the two instances. If you are still unable to find the instance, it could be a UI bug or some other issue. In such a scenario, it would be good to open a ticket with Microsoft to investigate further.

[eager_to_learn · Accepted solution]

Hi @Kaniz Fatma​ 

I have followed @Prabakar Ammeappin​ suggestion and tried checking all the logs but could not get any real insights from there.

We were talking to few more experts got two more suggestions :

a) Queue the jobs somehow and jobs to wait until instances are available.

b) Another suggestion was to delete the pool and try recreating the same.

As a is time consuming and we tried the option b and that has worked out for us, may be a system bug.

/Thanks

[Prabakar]

hi @Arun Manocha​ I understand that option A could be time-consuming. But it would have helped us understand the issue's real cause. You have resolved the issue by deleting the pool and recreating a new one. But are you sure the two instances showing as running on the pools were deleted? If it's a bug on UI to show instance as running, then option B is fine. But if two instances were actually running then it might add to the billing. So it would be worth checking the cause of this issue.

[eager_to_learn]

Yes @Prabakar Ammeappin​ it looked like á bug as it was having nothing in the logs and after deletion its completely gone, pool is fully available for work

[eager_to_learn]

@Kaniz Fatma​ / @Prabakar Ammeappin​ Any idea, how can we queue the jobs in the Resource pools, is it some setting which we need to switch on so the jobs are queued until instances are available or can you point some documentation for the same ?