The FirewallSetupException is thrown when Cluster Manager tries to allow communication to newly launched containers and the node can’t apply updated iptables rules. This occurs in the code path for allowCommunicationFromOldHostsToNewContainers during add-containers/upsize operations.

A very common underlying cause is the node daemon failing to write the temporary firewall rule file due to “No space left on device,” which prevents iptables-restore from applying the rules.

Common root causes seen

• The instance’s root volume is full (often due to archived log-daemon usage logs under /home/ubuntu/databricks/log-daemon/work/...), leading to “No space left on device” during firewall rule generation and apply.

• Node daemon RPC failures (e.g., “Got invalid response: 404”) from the instance can also cause inbound firewall updates to fail.

• In the same window you’ll often see cluster events like “Could not register new workers with running worker …” as the upsize/add-containers retries time out.

What you can do now (quick mitigation)

• Retry the upsize or restart the cluster to replace the affected instances with fresh VMs, which typically clears local disk/log conditions.

• If you can reach the instance, quickly check disk pressure:

  • Run df -h and look for the root device (e.g., /dev/xvda1) at 100% usage.
  • If confirmed, reduce/cleanup oversized log archives on that host or replace the instance.

Mitigations:

• Shorten Spark event log rollover to reduce pressure on local storage during long-running jobs or noisy clusters, e.g.:

  • spark.databricks.eventLog.rolloverIntervalSeconds=300.

• Cleaning up oversized log-daemon archives on the affected host(s) restores autoscaling and allows firewall rule updates to succeed.