SQL warehouse failing to start ( Please check network connectivity from the data plane to the control plane )

Cano
New Contributor III

‎01-12-2023 01:04 PM

Hi, 

My SQL warehouse is failing to start with the following error message:

Details for the latest failure: Error: [id: InstanceId(i-01b84b6705ff09104), status: INSTANCE_INITIALIZING, workerEnvId:WorkerEnvId(workerenv-3023557811934763-c8cef827-a038-4552-a74d-829ee273d683), lastStatusChangeTime: 1673489057531, groupIdOpt Some(0),requestIdOpt Some(0112-020302-ltb2xfyq-2b759b89-28f3-42ff-b),version 2] with threshold 700 seconds timed out after 704732 milliseconds. Please check network connectivity from the data plane to the control plane. Code: BOOTSTRAP_TIMEOUT Cluster-id (internal): 0112-020302-ltb2xfyq Additional details: {"instance_id":"i-01b84b6705ff09104"}

I followed the steps here https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#...

All the requirements for security group, subnet, NACLs, route table were met.

I even tried to create a new workspace but that also failed.

I have also attached the log file.

Your urgent help is hugely appreciated.

Labels:

Debayan
Databricks Employee
Databricks Employee

‎01-12-2023 01:44 PM

Hi, There is a line in the attached logs as below:

[Bootstrap Event] Can reach ohio.cloud.databricks.com: [FAILED]

[Bootstrap Event] DNS output for databricks-prod-artifacts-us-east-2.s3.us-east-2.amazonaws.com: 

Server: 10.187.0.2

Address: 10.187.0.2#53

This means somehow the webapp endpoint cannot reach as per the document: https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html?...

imageCould you please make sure if it has been added and let us know if it helps?

Preview file
213 KB

Cano
New Contributor III
In response to Debayan

‎01-13-2023 07:50 AM

Hi @Debayan Mukherjee​ 

Everything is managed through the security group and it allows all traffic. And there is also a NAT gateway associated to the subnet to provide internet connectivity.

I see the Databricks Instances spin up in my AWS account, is there Is anyway I can connect to these instances to a run a few tests?

I have attached a screenshot of both inbound and outbound rules for your viewing.

Please let me know if you have any other suggestions.

0 Kudos

Cano
New Contributor III
In response to Cano

‎01-13-2023 08:16 AM

Please note that I'm not using a customer managed VPC, but a VPC that was created by Databricks when I created the workspace.

0 Kudos

Debayan
Databricks Employee
Databricks Employee
In response to Cano

‎01-15-2023 08:56 PM

Hi @charles okoh​ , Thanks for all the details. I understand it is a default VPC. In that case, could you please raise a support case with Databricks?

0 Kudos