topic Re: Quota Limit Exhausted Error when Creating declarative pipeline in Data Engineering

Quota Limit Exhausted Error when Creating declarative pipeline

crami — Fri, 31 Oct 2025 15:43:48 GMT

I am trying to develop a declarative pipeline. As per platform policy, I cannot use serverless, reason, I am using asset bundle to create declarative pipeline. In the bundle, I am trying to specify compute for the pipeline. However, I am constantly facing quota exhausted exception.
Example: Below is compute config I am specifying..

Below is quota availability..

I have tried specifying DSv2 and DSv3 family cluster, all hit the same issue...
Is there any specific configuration I may need to specify for cluster to instantiate ?
Exception:

Re: Quota Limit Exhausted Error when Creating declarative pipeline

Khaja_Zaffer — Fri, 31 Oct 2025 16:35:55 GMT

Hello @crami

Good day!!

As the error tells. you need to increase the VM size, i know you have enough things in your place but

spot fallback + Photon + autoscale triggers the failure.

Go to

Azure Portal → Subscriptions →

Usage + quotas

Filter: Provider=Microsoft.Compute, Location=

Also, did you try with availability as "ON_DEMAND"?

Redeploy bundle: databricks bundle deploy Start pipeline. This might bypasses spot quota/capacity checks.

i am also looking for resolutions from other contributors.

Thank you.

Re: Quota Limit Exhausted Error when Creating declarative pipeline

crami — Sat, 01 Nov 2025 11:08:11 GMT

Thanks for pointer.