topic Re: Failed to launch pipeline cluster in Data Engineering

Failed to launch pipeline cluster

SimonXu — Fri, 02 Dec 2022 03:31:36 GMT

Hi, there. I encountered an issue when I was trying to create my delta live table pipeline.

The error is "DataPlaneException: Failed to launch pipeline cluster 1202-031220-urn0toj0: Could not launch cluster due to cloud provider failures. azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in exceeding approved standardFSF...".

However, when I checked my azure subscription, it showed that I had much enough quota space. I don't how to fix this issue as I'm new to the delta live table.

Re: Failed to launch pipeline cluster

UmaMahesh1 — Fri, 02 Dec 2022 07:53:07 GMT

Hi @Simon Xu

I hope this thread might solve your issue..

https://learn.microsoft.com/en-us/answers/questions/195003/databricks-cluster-error-code-operationnotallowed.html

Cheers

Re: Failed to launch pipeline cluster

Anonymous — Fri, 02 Dec 2022 08:01:23 GMT

Hi @Simon Xu,

You're not alone.

I encountered this issue before. The issue comes from Azure site not Databricks.

You need to check the number of cores, ram, CPU in your Warehouse cluster then compare the resources in Azure resource group hosted Databricks workspace then if you don't have

enough resource you need to increase the quota to higher number.

Feel free to ask me if you have any questions 😀

BR,

Jensen Nguyen

Re: Failed to launch pipeline cluster

arpit — Thu, 26 Jan 2023 19:20:38 GMT

@Simon Xu

I suspect that DLT is trying to grab some machine types that you simply have zero quota for in your Azure account. By default, below machine type gets requested behind the scenes for DLT:

AWS: c5.2xlarge

Azure: Standard_F8s

GCP: e2-standard-8

You can also set them explicitly from here.

Regards,

Arpit Khare

Re: Failed to launch pipeline cluster

SimonXu — Mon, 30 Jan 2023 23:18:57 GMT

Thanks, Jensen. It works for me!

Re: Failed to launch pipeline cluster

Adedotun — Thu, 30 Nov 2023 15:32:37 GMT

Unfortunately, I just encountered this error too, and followed your solution but it's still not working. My Usage + quota on Azure is 4 out of 10 (6) and the required DBs compute is 4 cores. However in my case, I used a single node. I strongly suspect I have to switch to a multi-node cluster, and then request for an increase in cores from Azure. I'll be back with an update!

Re: Failed to launch pipeline cluster

Masoomeh — Thu, 21 Dec 2023 10:26:24 GMT

I followed @arpit suggestion and set the cluster configuration explicitly in the JSON file and solved the issue.

Re: Failed to launch pipeline cluster

Yaadhu — Thu, 12 Dec 2024 19:59:16 GMT

you can create the pool instance in the databricks under compute/pool and assign the value in the json of the DLT pipeline. With this, we will control on pool min workers and max workers and the reuse of the pools available by other pipelines.

"node_type_id": "i4i.xlarge",
"driver_node_type_id": "i4i.2xlarge",
"instance_pool_id": "pool id from the pool configured in pools ",
"driver_instance_pool_id": "pool id from the pool configured in pools ",
"autoscale": {
"min_workers": 1,
"max_workers": 6,
"mode": "ENHANCED"
}