Facing CANNOT_OPEN_SOCKET error after job cluster fails to upsacle to target nodes

[ashishCh]

This error pops up in my Databricks workflow 1 out of 10 times, and everytime it occurs I see the below message in event logs.
 Compute upsize complete, but below target size. The current worker count is 1, out of a target of 3.


And right after this my job cluster terminates with the socket error message.

These are my cluster configs, if required.

[Coffee77]

Difficult to know but maybe it has to do with usage of spot instances as it seems root cause is kind of random. In theory spot instances can be terminated at any time by cloud provider if it needs the capacity back, BUT databricks should handle this fact correctly to replace lost spot workers or apply resilient policies to avoid that type of errors.

So, I can't ensure that is your issue. However, you can try to disable that option for a given time taking into account that costs will be a little higher. In anycase, don't use "spot" instances in PROD unless your workloads can afford breaks.