Cluster xxxxxxx was terminated during the run.

[Eduard]

Hello,

I have a problem with the autoscaling of a cluster. Every time the autoscaling is activated I get this error. Does anyone have any idea why this could be?

"Cluster xxxxxxx was terminated during the run (cluster state message: Lost communication with the driver node. This can occur because of networking errors or malfunctioning instances. databricks_error_message: driver is lost) "

Also from time to time I get this error also: 

Cluster xxxxxx  was terminated during the run (cluster state message: Setting up 6 nodes.)

[Eduard]

So i could see more in deep the logs and i got this:

CPU is not the problem. 

Caused by: com.databricks.backend.manager.instance.FirewallSetupException: Fail to setup inbound Firewall.

I got this error while the autoscaling was ON. Must be something with my network, not sure what..