04-13-2022 05:08 AM
Hi all,
I am migrating to Databricks E2 from older one. I moved the cluster definitions from the old databricks instance as well as creating new ones. Databricks tries to start a cluster for an hour and then fails. This happens for modes: Single Node and Standard.
I have checked out this topic as well https://community.databricks.com/s/question/0D53f00001in5HDCAY/databricks-cluster-create-fail
but without any luck.
I don't see any AWS quotas to be reached.
Cluster terminated.Reason:Unexpected launch failure
An unexpected error was encountered while setting up the cluster. Please retry and contact Databricks if the problem persists.
Internal error message: java.lang.RuntimeException: Internal error (no failure to report) at com.databricks.backend.manager.AddResourcesStateHelper$.<init>(AddResourcesState.scala:216) at com.databricks.backend.manager.AddResourcesStateHelper$.<clinit>(AddResourcesState.scala) at com.databricks.backend.manager.ClusterManager.shouldStopAddingNodes(ClusterManager.scala:3979) at com.databricks.backend.manager.ClusterManager.runAddResourceSteps(ClusterManager.scala:4125) at com.databricks.backend.manager.ClusterManager.addResourcesToCluster(ClusterManager.scala:4033) at com.databricks.backend.manager.ClusterManager.$anonfun$doAddContainersToCluster$1(ClusterManager.scala:2158) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:366) at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:460) at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:480) at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$2(UsageLogging.scala:232) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:94) at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:230) at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:212) at com.databricks.backend.manager.ClusterManager.withAttributionContext(ClusterManager.scala:147) at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276) at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:261) at com.databricks.backend.manager.ClusterManager.withAttributionTags(ClusterManager.scala:147) at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:455) at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:375) at com.databricks.backend.manager.ClusterManager.recordOperationWithResultTags(ClusterManager.scala:147) at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:366) at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338) at com.databricks.backend.manager.ClusterManager.recordOperation(ClusterManager.scala:147) at com.databricks.backend.manager.ClusterManager.doAddContainersToCluster(ClusterManager.scala:2158) at com.databricks.backend.manager.ClusterManager.$anonfun$doSetupCluster$3(ClusterManager.scala:542) at com.databricks.backend.manager.ClusterManager.withAuditLog(ClusterManager.scala:2578) at com.databricks.backend.manager.ClusterManager.$anonfun$doSetupCluster$2(ClusterManager.scala:496) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:366) at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:460) at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:480) at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$2(UsageLogging.scala:232) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:94) at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:230) at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:212) at com.databricks.backend.manager.ClusterManager.withAttributionContext(ClusterManager.scala:147) at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276) at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:261) at com.databricks.backend.manager.ClusterManager.withAttributionTags(ClusterManager.scala:147) at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:455) at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:375) at com.databricks.backend.manager.ClusterManager.recordOperationWithResultTags(ClusterManager.scala:147) at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:366) at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338) at com.databricks.backend.manager.ClusterManager.recordOperation(ClusterManager.scala:147) at com.databricks.backend.manager.ClusterManager.$anonfun$doSetupCluster$1(ClusterManager.scala:496) at com.databricks.backend.manager.ClusterManager.catchInternalErrors(ClusterManager.scala:2605) at com.databricks.backend.manager.ClusterManager.doSetupCluster(ClusterManager.scala:478) at com.databricks.backend.manager.ClusterManager.doSetupOrUpsize(ClusterManager.scala:2771) at com.databricks.backend.manager.UpsizeThrottlingMonitor.$anonfun$processRequest$3(UpsizeThrottlingMonitor.scala:363) at com.databricks.backend.manager.util.ConsolidatedClusterUpdateHelper.$anonfun$withConsolidatedClusterUpdateForAsync$1(ConsolidatedClusterUpdateHelper.scala:142) at scala.util.Try$.apply(Try.scala:213) at com.databricks.backend.manager.util.ConsolidatedClusterUpdateHelper.withConsolidatedClusterUpdate(ConsolidatedClusterUpdateHelper.scala:61) at com.databricks.backend.manager.util.ConsolidatedClusterUpdateHelper.withConsolidatedClusterUpdateForAsync(ConsolidatedClusterUpdateHelper.scala:141) at com.databricks.backend.manager.util.ConsolidatedClusterUpdateHelper.withConsolidatedClusterUpdateForAsync$(ConsolidatedClusterUpdateHelper.scala:135) at com.databricks.backend.manager.UpsizeThrottlingMonitor.withConsolidatedClusterUpdateForAsync(UpsizeThrottlingMonitor.scala:76) at com.databricks.backend.manager.UpsizeThrottlingMonitor.$anonfun$processRequest$2(UpsizeThrottlingMonitor.scala:363) at
Thank you for your support
04-15-2022 04:22 AM
Please check:
04-15-2022 04:22 AM
Please check:
04-25-2022 07:27 AM
Thanks for your reply.
Indeed, I've been facing networking issue - your hint was very helpful!
04-25-2022 10:33 AM
Good to hear that it helped. If you can, please select my answer as the best one.
04-26-2022 03:43 AM
Thank you @Hubert Dudek for your fantastic response.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group