Spark doesn't register executors when new workers are allocated

[ivanychev]

Our pipelines sometimes get stuck (example).

Some workers get decommissioned due to spot termination and then the new workers get added.

 However, after (1) Spark doesn't notice new executors:

 And I don't know why. I don't understand how to debug this, but here're some of my observations:

* The init script logs of the workers, which Spark doesn't notice, are fine, they complete successfully.

* The driver logs don't show anything significant after old executors get decomissioned. Driver simply doesn't notice new executors

How do I  debug this and what can be the issue?

[shan_chandra]

@ivanychev  - Firstly, New workers are added and spark notice them hence, there is an init script logging in the event log stating the init script ran on the newly added workers.  For debugging, please check the Spark UI - executor tab. 

Secondly, For Spot Instance termination, This is mostly by the cloud provider and spot instance price fluctuation. you can ideally use hybrid clusters (with spot fall back on demand) flag set on the cluster configuration page. 

Reference: https://docs.databricks.com/en/compute/cluster-config-best-practices.html#on-demand-and-spot-instanc...

Thanks, Shan