Hi @Raghav1,

Thanks for the detailed write-up. This is a helpful description of what you’re seeing.

From what you shared, there are a couple of different layers in play here:

• Azure Free Trial quotas/VM availability (the QuotaExceeded and "VM size not available" errors)
• Databricks configuration (how the DLT pipeline compute is defined and how the cluster policy is applied)

Right now, it’s hard to tell whether the main issue is that the pipeline isn’t honouring the policy or that Azure simply can’t provision the requested SKU/cores even if the config is valid. To narrow this down, can you share a bit more detail?

Pipeline definition / CLI command: Could you paste the JSON you’re using with databricks pipelines create (or the equivalent API payload), especially the parts that define:

• cluster_spec / development/production compute
• Any policy_id you’re setting
• Explicit node_type_id / instance_pool_id
• Min / max workers

If you’re using Databricks pipelines, edit or similar via the CLI, please include that command and the JSON as well (with any secrets removed).

Cluster policy JSON: Please also share the relevant parts of the cluster policy you created, e.g.:

• Whether you’re:

  • Forcing a specific small VM (e.g., via defaultValue + fixed), or
  • Only constraining ranges / families (allowed_values, regex, etc.)

• Any rules related to:

  • node_type_id / driver_node_type_id
  • num_workers or autoscaling
  • Pools vs on-demand VMs

This helps determine whether the policy is just constraining values or is actually designed to pin the pipeline to a smaller SKU that fits your Free Trial quota.

A couple of clarifications / expectations

• Cluster policies constrain what can be configured. They don’t automatically downshift a pipeline to the smallest possible VM. If the pipeline definition is explicitly using Standard_F4s, and that value is still allowed by the policy, Databricks won’t silently swap it for a smaller SKU... the pipeline compute block usually needs to be set to that smaller instance.
• The error Modifying following parameter gateway_definition in pipeline settings is not allowed indicates that some parts of the pipeline definition are treated as immutable after creation. In practice this often means:
  • To change certain compute aspects, you may need to recreate the pipeline with the desired compute/policy settings.
  • Or adjust what’s controlled by the policy vs what’s hard-coded in the pipeline.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.