NPIP tunnel setup failure during launch

lgepp11 · ‎09-25-2023

In AWS with this current error when spinning up the SQL warehouse or personal compute.
Backend private link is enabled.

Error: NPIP tunnel setup failure during launch. Please try again later and contact Databricks if the problem persists. Instance bootstrap failed command: WaitForNgrokTunnel Failure message (may be truncated): Timed out waiting for ngrok tunnel to be up Code: NPIP_TUNNEL_SETUP_FAILURE

On the EC2 instance I can see this failure in the logs:
2023/09/25 02:36:27 INFO vm_bootstrap.py:1118: [Bootstrap Event] Command BuiltInCommand_WaitForNgrokTunnel finished. [FAILED] . Seconds Elapsed: 300.273530006

2023/09/25 02:36:27 INFO vm_bootstrap.py:230: [Bootstrap Event] {FAILED_COMMAND:WaitForNgrokTunnel}

2023/09/25 02:36:27 INFO vm_bootstrap.py:232: [Bootstrap Event] {FAILED_MESSAGE:(Base64 encoded) VGltZWQgb3V0IHdhaXRpbmcgZm9yIG5ncm9rIHR1bm5lbCB0byBiZSB1cA== }

2023/09/25 02:36:27 INFO vm_bootstrap.py:1109: [Bootstrap Event] Command GetInstanceId finished. [SUCCEEDED]. Seconds Elapsed: 0.00336885452271

2023/09/25 02:36:27 INFO vm_bootstrap.py:1109: [Bootstrap Event] Command UpdateBootstrapStatus finished. [SUCCEEDED]. Seconds Elapsed: 0.0854570865631

User16539034020 · ‎09-28-2023

Hello,

Thanks for contacting Databricks Support.

Based on the error message: NPIP_TUNNEL_SETUP_FAILURE. It indicates that bootstrap failed due to network connectivity issues between the data plane and control plane.

Seems like you have already download the system log, please use Base64 decode tool to decode FAILED_MESSAGE. "2023/09/25 02:36:27 INFO vm_bootstrap.py:232: [Bootstrap Event] {FAILED_MESSAGE:(Base64 encoded) VGltZWQgb3V0IHdhaXRpbmcgZm9yIG5ncm9rIHR1bm5lbCB0byBiZSB1cA== }", you will get the RCA for the failure.

lgepp11 · ‎12-20-2023

Timed out waiting for ngrok tunnel to be up.....

This doesn't tell me anything about what could be causing the issue though? What else can I check? has anyone else seen this before?