topic Re: Databricks cluster start-up: Self Bootstrap Failure in Data Engineering

Databricks cluster start-up: Self Bootstrap Failure

mikaellognseth — Tue, 21 Jun 2022 09:48:06 GMT

When attempting to deploy/start an Azure Databricks cluster through the UI, the following error consistently occurs:

{
  "reason": {
    "code": "SELF_BOOTSTRAP_FAILURE",
    "parameters": {
      "databricks_error_message": "Self-bootstrap failure during launch. Please try again later and contact Databricks if the problem persists. Node daemon fast failed and did not answer ping for instance a3488320d27a401fa679089b87410fab"
    }
  }
}

When looking at the resource group, I can see that the resources are being deployed, but as the self-bootstrap step fails, they are removed again.

We're using VNET injection with two /26 subnets for the Databricks workspace. Both subnets have the "Microsoft.Databricks/workspaces" service delegation and the following service delegation actions:

"Microsoft.Network/virtualNetworks/subnets/join/action",

"Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action", "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"

All of the standard NSG rules are added and seem correct when comparing to the official Microsoft documentation.

We are using Terraform to deploy, I have added the relevant bits of the DBW deployment here.

Re: Databricks cluster start-up: Self Bootstrap Failure

mikaellognseth — Thu, 30 Jun 2022 06:07:48 GMT

Hi,

in our case the issue turned out to be DNS... As the DNS servers set on the Databricks workspace vnet are only available when peering the "management" vnet in our setup. Took a while to figure out as the error didn't exactly give a lot of clarity. 🙂

Re: Databricks cluster start-up: Self Bootstrap Failure

AJK1 — Tue, 13 Dec 2022 16:55:43 GMT

Hello.

I'm getting the same error as well. I'm also pretty certain that we setup the workspace correctly within our vNet, per Microsoft instructions. May I ask you - how did you specifically go about solving the DNS issue?

Re: Databricks cluster start-up: Self Bootstrap Failure

Dooley — Tue, 13 Dec 2022 17:56:49 GMT

@Andrew Kwiatkowski Did you also set up your workspace with Terraform? Did you never get a cluster to run in your new workspace or has it worked before and now it doesn't work?

Re: Databricks cluster start-up: Self Bootstrap Failure

AJK1 — Tue, 13 Dec 2022 20:24:19 GMT

Hi Sara. Apologies – I should have specified… we do not use Terraform. I was never able to actually start a cluster successfully, just define it.

Re: Databricks cluster start-up: Self Bootstrap Failure

Dooley — Tue, 13 Dec 2022 23:09:19 GMT

Have you resolved your DNS for the newly created Azure workspace? Here are the Azure instructions on how to troubleshoot your DNS.

Re: Databricks cluster start-up: Self Bootstrap Failure

Dooley — Tue, 13 Dec 2022 23:11:37 GMT

Also are you using Databricks PrivateLink?

Re: Databricks cluster start-up: Self Bootstrap Failure

AJK1 — Wed, 14 Dec 2022 18:27:44 GMT

Yes, we’re attempting to follow the steps outlined for PrivateLink.