We hit this exact failure mode a few months ago, so I can share how it played out for us and where I think your situation differs.
Observations: Every cluster in one of our environments started failing with BOOTSTRAP_TIMEOUT. The instances themselves launched fine, we could see them running on the cloud side, but the Databricks bootstrap agent on the node never registered with the control plane, and after the 700 second threshold each cluster gave up and terminated. Nothing in our code was involved, since the cluster never got far enough to run any of it.
Root cause: The egress firewall only allowed port 443 to the Databricks control plane. The bootstrap agent also needs a set of additional ports for its registration call, all documented in the Databricks networking requirements, and those rules were missing. The interesting part is how they went missing: our IAC defined the full rule set, but the live firewall had drifted and only carried one rule. A second environment with the full rule set worked fine the whole time, and comparing the two firewalls side by side is what exposed it.
Resolution: We re-applied the infrastructure code so the live firewall matched the full documented list of ports and endpoints. Clusters came up within a few minutes of the apply. We also added firewall logging afterward, because without it we had to infer the dropped traffic from rule comparison rather than seeing it directly.
The issue that you are experiencing is similar to ours but not identical. Ours was on AWS and the blocked traffic was the control plane registration call. Yours is on Azure, and your screenshot shows the failure earlier in the sequence: Command.downloadNodeSetupFiles timing out at about 168 seconds per attempt, which means the node cannot pull its setup files from the Databricks artifact storage endpoints.
Your error is also classified CLIENT_ERROR, which is Databricks telling you their telemetry points at your network path, so the platform agrees with Azure support on where the problem lives. But the underlying pattern looks the same as ours: an egress allowlist that is partially right, and the tell in both cases is that a fix helps some clusters while others keep failing.
Here are some steps that you can consider pursuing:
1. Your network table says "including default artifact storage." Each Azure region has more than one artifact storage account, and the full list is in the Azure Databricks user-defined route settings documentation. Which account a node pulls from can vary with the runtime version. So compare the runtime version of your all-purpose clusters against your job clusters. If they differ, and your rules only cover the default artifact account, that alone would produce exactly the split you're seeing. Allow every artifact storage FQDN listed for your region, not just the default.
2. Job clusters pay the download on every run, and all workers pull at once. An all-purpose cluster downloads once at startup and then sits warm, so after a partial fix it looks healthy. A scheduled multi-node job provisions fresh VMs each run, each pulling large files through your firewall at the same moment. If egress goes through Azure Firewall or a NAT gateway, that burst is where SNAT port exhaustion and throughput ceilings show up. Check firewall metrics and SNAT utilization at the timestamps the job clusters fail.
3. Confirm the updated rules cover the subnets the job clusters actually use. Your table shows different source ranges for different services. If the job compute runs in a different workspace or subnet pair than the all-purpose clusters you tested, the improvement may simply not reach it. This is the shape our incident took, rules correct in one place and stale in another, so I would not skip this check even if it feels obvious.
For diagnosis beyond the UI message, pull the cluster events for a failed job cluster:
databricks clusters events <cluster_id>
The ADD_NODES_FAILED events carry the detailed download log, which tells you whether the transfer connected and crawled (throttling or saturation) or never connected at all (an endpoint still blocked). Those are different fixes, so it's worth knowing which one you have before changing anything else.
Hope it helps!
