https://community.databricks.com/t5/machine-learning/intermittent-quot-unexpected-failure-while-waiting-for-the/m-p/20920#M1153 <P>Thank you for your response! We are on AWS rather than Azure, and (as a result?) the error messages do not seem to match those in the article you provide. We have tried some of the suggestions anyways, such as removing external maven libraries, to no avail.</P><P></P><P>Switching to pools has mostly reduced the issue, but when new clusters are needed because the pool clusters are already in use, we occasionally receive the original error still.</P> Wed, 08 Jun 2022 16:56:03 GMT 108387 2022-06-08T16:56:03Z https://community.databricks.com/t5/machine-learning/intermittent-quot-unexpected-failure-while-waiting-for-the/m-p/20918#M1151 <P>We are encountering an occasional issue where jobs may fail with a message like the following:</P><P></P><P>Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster (ID) to be ready.Cause Unexpected state for cluster (ID): BOOTSTRAP_TIMEOUT(SUCCESS): databricks_error_message:[id: InstanceId(ID), status: INSTANCE_INITIALIZING, workerEnvId:WorkerEnvId(ID), lastStatusChangeTime: 1651979481336, groupIdOpt None,requestIdOpt Some(ID),version 0] with threshold 700 seconds timed out after 700186 milliseconds. Please check network connectivity from the data plane to the control plane.,instance_id:ID</P><P></P><P>We have seen the related post <A href="https://community.databricks.com/s/question/0D53f00001fR8LGCA0/problem-with-spinning-up-a-cluster-on-a-new-workspace" target="test_blank">https://community.databricks.com/s/question/0D53f00001fR8LGCA0/problem-with-spinning-up-a-cluster-on-a-new-workspace</A>, but unlike that issue, ours fails less than 5% of the time. Any job may fail due to this issue, and there is no common time for this to happen.</P><P></P><P>We were able to pull AWS EC2 logs of failed and successful runs, but there are no obvious errors or differences between the two. The failed runs still bootstrap correctly and connect to Databricks, for instance.</P><P></P><P>If it helps, all jobs are set up through dbx using the same cluster settings (spot with fallback).</P><P></P><P>What exactly does this error mean, and how might we go about addressing it?</P> Tue, 10 May 2022 18:20:49 GMT https://community.databricks.com/t5/machine-learning/intermittent-quot-unexpected-failure-while-waiting-for-the/m-p/20918#M1151 108387 2022-05-10T18:20:49Z https://community.databricks.com/t5/machine-learning/intermittent-quot-unexpected-failure-while-waiting-for-the/m-p/20920#M1153 <P>Thank you for your response! We are on AWS rather than Azure, and (as a result?) the error messages do not seem to match those in the article you provide. We have tried some of the suggestions anyways, such as removing external maven libraries, to no avail.</P><P></P><P>Switching to pools has mostly reduced the issue, but when new clusters are needed because the pool clusters are already in use, we occasionally receive the original error still.</P> Wed, 08 Jun 2022 16:56:03 GMT https://community.databricks.com/t5/machine-learning/intermittent-quot-unexpected-failure-while-waiting-for-the/m-p/20920#M1153 108387 2022-06-08T16:56:03Z