Ashwin_DSA
Databricks Employee
Databricks Employee

‎03-24-2026 12:32 AM

Hi @Neelimak,

For managed ingestion pipelines, the auto‑generated cluster is just a classic jobs cluster whose shape is controlled by a compute policy, so you can override the VM type and add fallbacks.

Ask a workspace admin to create or edit a Job Compute / cluster_type = dlt policy and pin it to VM sizes that actually exist in your region, for example:

 

{
  "cluster_type": { "type": "fixed", "value": "dlt" },
  "num_workers": { "type": "unlimited", "defaultValue": 1, "isOptional": true },
  "driver_node_type_id": { "type": "fixed", "value": "REGION_SUPPORTED_DRIVER" },
  "node_type_id": { "type": "fixed", "value": "REGION_SUPPORTED_WORKER" }
}
Then make sure the ingestion gateway uses this policy (via the policy family or explicit assignment).
 
To get automatic fallback to alternative VM types when that SKU hits capacity or quota, enable flexible node types for the workspace and, if needed, set driver_node_type_flexibility.alternate_node_type_ids and worker_node_type_flexibility.alternate_node_type_ids in the policy (comma‑separated list of compatible SKUs).

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***