06-02-2022 07:30 AM
We are using Azure Databricks pools, configured 16 max instances. Out of 16, 2 instances are in running state without any job in running condition, how & where can i check the usage of the instances ?
p.s. SQL pool is also not running, so no chances of instances being consumed there.
06-09-2022 01:45 AM
Hi @Kaniz Fatma
I have followed @Prabakar Ammeappin suggestion and tried checking all the logs but could not get any real insights from there.
We were talking to few more experts got two more suggestions :
a) Queue the jobs somehow and jobs to wait until instances are available.
b) Another suggestion was to delete the pool and try recreating the same.
As a is time consuming and we tried the option b and that has worked out for us, may be a system bug.
/Thanks
06-02-2022 08:44 AM
06-02-2022 08:52 AM
Hi @Prabakar Ammeappin
No, we have kept the idle as 0 and moreover its keeping 2 jobs always running which is causing capacity constraint for us and its getting difficult to find out where its running.
06-02-2022 11:10 AM
@Arun Manocha, You can check it in the Azure portal. I believe the running instances should have the pool name tagged. So you can do a backtrace and check the cluster tag for that instance. Then take the cluster-ID to the Databricks clusters page and use the cluster-ID in the cluster URL to get the running cluster. Once you get the group, you can find what is using the two instances. If you are still unable to find the instance, it could be a UI bug or some other issue. In such a scenario, it would be good to open a ticket with Microsoft to investigate further.
06-09-2022 12:35 AM
Hi @Arun Manocha , We haven’t heard from you on the last response from @Prabakar Ammeappin , and I was checking back to see if you have a resolution yet.
Were you able to follow @Prabakar Ammeappin's suggestions? Did you open any tickets?
Or else, If you have any solution, please do share that with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.
06-09-2022 01:45 AM
Hi @Kaniz Fatma
I have followed @Prabakar Ammeappin suggestion and tried checking all the logs but could not get any real insights from there.
We were talking to few more experts got two more suggestions :
a) Queue the jobs somehow and jobs to wait until instances are available.
b) Another suggestion was to delete the pool and try recreating the same.
As a is time consuming and we tried the option b and that has worked out for us, may be a system bug.
/Thanks
06-09-2022 04:28 AM
hi @Arun Manocha I understand that option A could be time-consuming. But it would have helped us understand the issue's real cause. You have resolved the issue by deleting the pool and recreating a new one. But are you sure the two instances showing as running on the pools were deleted? If it's a bug on UI to show instance as running, then option B is fine. But if two instances were actually running then it might add to the billing. So it would be worth checking the cause of this issue.
06-10-2022 05:06 AM
Yes @Prabakar Ammeappin it looked like á bug as it was having nothing in the logs and after deletion its completely gone, pool is fully available for work
06-10-2022 05:14 AM
Cool!
Thank you for your update @Arun Manocha!
I'm glad it worked.
06-10-2022 06:12 AM
@Kaniz Fatma / @Prabakar Ammeappin Any idea, how can we queue the jobs in the Resource pools, is it some setting which we need to switch on so the jobs are queued until instances are available or can you point some documentation for the same ?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.