User Initiated Cluster Termination, after 1-2 hours cluster goes offline even while working!

[gazzyjuruj]

Hi,

Wanted to ask, I have been experiencing

databricks_error = user initiated cluster termination

databricks_error_message: timeout while placing nodes

and soetimes it stays stuck on "finding new nodes, instances if neccessary" for hours..

and it goes offline usually after 1-2 hours while working, what is causing this? How do we fix it?

Thank you, have a great day ahead.

[Kaniz]

Hi @Ghazanfar Uruj​ , Are you still facing the issue?

[gazzyjuruj]

Hi yes, it does go offline "connection closed" in case im using terminal even faster maybe only in 30 minutes.

and in 2 hours it goes off, i know the idle timeout limits, but my processes are running while it goes down.

and on the Client.UserInitiatedShutdown error im again running my applications and i will update you in the next 1-2 hours. Thanks @Kaniz Fatma​ 

[Kaniz]

Thank you for your quick response @Ghazanfar Uruj​ . Please keep us updated.

[Kaniz]

Hi @Ghazanfar Uruj​ , Did you try on the Client.UserInitiatedShutdown error?

[Atanu]

We have identified that the recent deployment done on the backend service on which Azure Data factory relies upon had prevented the data flow activity from running and this had caused the issue

But can you confirm @Ghazanfar Uruj​ if the issue persists . also the timeline would help.

Thanks

[Kaniz]

Hi @Ghazanfar Uruj​  , We haven’t heard from you on the last response from @Atanu Sarkar​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.