topic Re: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster in Data Engineering

CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster

SrinuM — Wed, 26 Jun 2024 10:21:54 GMT

One of our databricks workflow job is failing occasionally with below error, after re-running and working fine without any issue.

What is the exact reason for the issue and how can we fix it

Error:

Unexpected failure while waiting for the cluster to be ready: Cluster 'XXX' was terminated. Reason: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE). Parameters: azure_error_code:StorageFailure, azure_error_message:Error while creating storage object'

Re: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster

jacovangelder — Wed, 26 Jun 2024 10:51:40 GMT

What cloud are you on? Azure? The error from my experience indicate either one of the following:

There was a temporary cloud outtage/downtime
Bust most likely: you've reached your cloud subscriptions VM/CPU quota. Most likely you need to increase the subscription quota for the VM type you're using.

Re: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster

SrinuM — Wed, 26 Jun 2024 11:46:40 GMT

Its Azure, we have the VM/CPU quota available, i don't think that is causing the issue

Re: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster

jacovangelder — Wed, 26 Jun 2024 11:54:31 GMT

The problem with this error is that its very hard to debug.
I would create an Azure support ticket so they can look into the actual cause.

Re: CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster

PSR100 — Thu, 11 Jul 2024 09:06:38 GMT

These are cloud provider related errors and we will not have much error details from the error message. Based on the error message and also, that you have enough CPU/VM quota available, I think the issue might be due to the storage creation stage in the cluster startup.

If the workspace is a VNET injected workspace, you can try adding Microsoft.Storage service endpoint for both the public and private subnets used for the workspace and check if you are still seeing the same behaviour.