โ06-02-2022 07:30 AM
We are using Azure Databricks pools, configured 16 max instances. Out of 16, 2 instances are in running state without any job in running condition, how & where can i check the usage of the instances ?
p.s. SQL pool is also not running, so no chances of instances being consumed there.
โ06-09-2022 01:45 AM
Hi @Kaniz Fatmaโ
I have followed @Prabakar Ammeappinโ suggestion and tried checking all the logs but could not get any real insights from there.
We were talking to few more experts got two more suggestions :
a) Queue the jobs somehow and jobs to wait until instances are available.
b) Another suggestion was to delete the pool and try recreating the same.
As a is time consuming and we tried the option b and that has worked out for us, may be a system bug.
/Thanks
โ06-02-2022 08:44 AM
hi @Arun Manochaโ It could be possible that you have min idle set to 2. This is the minimum number of instances the pool will keep idle. These instances do not terminate, regardless of the setting specified in Idle Instance Auto Termination.
โ06-02-2022 08:52 AM
Hi @Prabakar Ammeappinโ
No, we have kept the idle as 0 and moreover its keeping 2 jobs always running which is causing capacity constraint for us and its getting difficult to find out where its running.
โ06-02-2022 11:10 AM
@Arun Manochaโ, You can check it in the Azure portal. I believe the running instances should have the pool name tagged. So you can do a backtrace and check the cluster tag for that instance. Then take the cluster-ID to the Databricks clusters page and use the cluster-ID in the cluster URL to get the running cluster. Once you get the group, you can find what is using the two instances. If you are still unable to find the instance, it could be a UI bug or some other issue. In such a scenario, it would be good to open a ticket with Microsoft to investigate further.
โ06-09-2022 01:45 AM
Hi @Kaniz Fatmaโ
I have followed @Prabakar Ammeappinโ suggestion and tried checking all the logs but could not get any real insights from there.
We were talking to few more experts got two more suggestions :
a) Queue the jobs somehow and jobs to wait until instances are available.
b) Another suggestion was to delete the pool and try recreating the same.
As a is time consuming and we tried the option b and that has worked out for us, may be a system bug.
/Thanks
โ06-09-2022 04:28 AM
hi @Arun Manochaโ I understand that option A could be time-consuming. But it would have helped us understand the issue's real cause. You have resolved the issue by deleting the pool and recreating a new one. But are you sure the two instances showing as running on the pools were deleted? If it's a bug on UI to show instance as running, then option B is fine. But if two instances were actually running then it might add to the billing. So it would be worth checking the cause of this issue.
โ06-10-2022 05:06 AM
Yes @Prabakar Ammeappinโ it looked like รก bug as it was having nothing in the logs and after deletion its completely gone, pool is fully available for work
โ06-10-2022 06:12 AM
@Kaniz Fatmaโ / @Prabakar Ammeappinโ Any idea, how can we queue the jobs in the Resource pools, is it some setting which we need to switch on so the jobs are queued until instances are available or can you point some documentation for the same ?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group