Updating DLT pipeline's cluster policy using Databricks CLI

[Raghav1]

**Subject:** Azure Databricks Pipeline Not Applying Cluster Policy and Failing with VM Quota Error

**Summary of Issue**

I am attempting to create and run a pipeline in Azure Databricks using a custom cluster policy to restrict compute resources due to Azure Free Trial subscription limits. However, the pipeline is not honoring the configured cluster policy and instead defaults to a larger compute configuration, which results in VM quota errors during cluster launch.

**Details**

* Environment: Azure Databricks (Free Trial Subscription on Azure)
* Objective: Create a pipeline with restricted compute using a custom cluster policy.
* Policy Configuration: A cluster policy was created to enforce smaller VM sizes and limited worker scaling.
* Pipeline Creation: The pipeline was created after defining the cluster policy.

**Observed Behavior**

1. The pipeline does not apply the configured cluster policy during creation.
2. Instead, the pipeline defaults to a compute configuration similar to:

* Worker type: Standard_F4s
* Autoscaling: Min workers = 1, Max workers = 5
3. Because the Azure Free Trial subscription has limited CPU quota, the cluster fails to start with the following error:

```
Error class: CLUSTER_LAUNCH_CLIENT_ERROR
SQL state: KDL01

Encountered Quota Exhaustion issue in your account:
The VM size you are specifying is not available.
QuotaExceeded: Operation could not be completed due to insufficient CPU quota.
```

4. Even after deleting the pipeline and recreating it, the cluster policy is still not being applied.
5. When attempting to update the pipeline configuration via CLI, another error occurs:

```
Modifying following parameter gateway_definition in pipeline settings is not allowed
```

This prevents modifying the pipeline compute configuration once it is created.

**Impact**

* The pipeline cannot start due to VM quota limitations.
* The custom cluster policy intended to enforce smaller compute resources is not being applied.
* Pipeline configuration cannot be updated due to the gateway_definition restriction.

**Expected Behavior**

* The pipeline should respect the configured cluster policy during creation.
* Compute configuration should align with the policy limits (smaller VM and restricted worker scaling).
* The pipeline cluster should launch successfully within the available Azure quota.

**Request**

Assistance is requested to determine:

1. Why the pipeline is not applying the configured cluster policy.
2. How to ensure pipelines respect cluster policy constraints during creation.
3. Recommended configuration for running pipelines within Azure Free Trial compute quotas.

[Ashwin_DSA]

Hi @Raghav1,

Thanks for the detailed write-up. This is a helpful description of what you’re seeing.

From what you shared, there are a couple of different layers in play here:

• Azure Free Trial quotas/VM availability (the QuotaExceeded and "VM size not available" errors)
• Databricks configuration (how the DLT pipeline compute is defined and how the cluster policy is applied)

Right now, it’s hard to tell whether the main issue is that the pipeline isn’t honouring the policy or that Azure simply can’t provision the requested SKU/cores even if the config is valid. To narrow this down, can you share a bit more detail?

Pipeline definition / CLI command: Could you paste the JSON you’re using with databricks pipelines create (or the equivalent API payload), especially the parts that define:

• cluster_spec / development/production compute
• Any policy_id you’re setting
• Explicit node_type_id / instance_pool_id
• Min / max workers

If you’re using Databricks pipelines, edit or similar via the CLI, please include that command and the JSON as well (with any secrets removed).

Cluster policy JSON: Please also share the relevant parts of the cluster policy you created, e.g.:

• Whether you’re:

  • Forcing a specific small VM (e.g., via defaultValue + fixed), or
  • Only constraining ranges / families (allowed_values, regex, etc.)

• Any rules related to:

  • node_type_id / driver_node_type_id
  • num_workers or autoscaling
  • Pools vs on-demand VMs

This helps determine whether the policy is just constraining values or is actually designed to pin the pipeline to a smaller SKU that fits your Free Trial quota.

A couple of clarifications / expectations

• Cluster policies constrain what can be configured. They don’t automatically downshift a pipeline to the smallest possible VM. If the pipeline definition is explicitly using Standard_F4s, and that value is still allowed by the policy, Databricks won’t silently swap it for a smaller SKU... the pipeline compute block usually needs to be set to that smaller instance.
• The error Modifying following parameter gateway_definition in pipeline settings is not allowed indicates that some parts of the pipeline definition are treated as immutable after creation. In practice this often means:
  • To change certain compute aspects, you may need to recreate the pipeline with the desired compute/policy settings.
  • Or adjust what’s controlled by the policy vs what’s hard-coded in the pipeline.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[SteveOstrowski]

Hi @Raghav1,

There are a few things happening here, so let me walk through each one and give you a path forward.

UNDERSTANDING THE GATEWAY_DEFINITION ERROR

The error "Modifying following parameter gateway_definition in pipeline settings is not allowed" occurs because certain pipeline parameters are treated as immutable once the pipeline is created. The gateway_definition is an internal field tied to how the pipeline's compute is initially provisioned, and it cannot be changed via a CLI update command after creation.

The recommended approach is to delete the pipeline and recreate it with the correct compute configuration from the start.

HOW TO PROPERLY ATTACH A CLUSTER POLICY TO A PIPELINE

When creating a Lakeflow Spark Declarative Pipeline (SDP) via the Databricks CLI or REST API, you need to include the policy_id in the clusters array of your pipeline definition JSON, along with apply_policy_default_values set to true. This ensures the policy defaults are applied to the pipeline compute.

Here is an example pipeline creation JSON that references a cluster policy:

{
"name": "my-sdp-pipeline",
"clusters": [
{
"label": "default",
"policy_id": "<your-cluster-policy-id>",
"apply_policy_default_values": true
}
],
"libraries": [
{
"notebook": {
"path": "/path/to/your/notebook"
}
}
],
"development": true
}

The "label": "default" ensures the policy applies to both the update cluster and the maintenance cluster that every pipeline creates.

To create this pipeline via the Databricks CLI:

databricks pipelines create --json '{
"name": "my-sdp-pipeline",
"clusters": [
{
"label": "default",
"policy_id": "<your-cluster-policy-id>",
"apply_policy_default_values": true
}
],
"libraries": [
{
"notebook": {
"path": "/path/to/your/notebook"
}
}
],
"development": true
}'

IMPORTANT: POLICIES CONSTRAIN, THEY DO NOT AUTO-DOWNSIZE

A cluster policy constrains what values are allowed for compute configuration. It does not automatically select the smallest possible VM for you. If your pipeline definition does not explicitly set a node_type_id, Databricks will auto-select one, and that auto-selected type must still comply with your policy. If the auto-selected type happens to be Standard_F4s and that exceeds your Azure Free Trial quota, the cluster will fail to launch.

To ensure a small enough VM is used, you should either:

1. Set a fixed node_type_id directly in the pipeline's clusters configuration:

{
"clusters": [
{
"label": "default",
"policy_id": "<your-policy-id>",
"apply_policy_default_values": true,
"node_type_id": "Standard_DS3_v2",
"driver_node_type_id": "Standard_DS3_v2",
"num_workers": 0
}
]
}

2. Or set a fixed value in the cluster policy itself so that any pipeline using it always gets a small VM:

{
"cluster_type": {
"type": "fixed",
"value": "dlt"
},
"node_type_id": {
"type": "fixed",
"value": "Standard_DS3_v2",
"hidden": true
},
"driver_node_type_id": {
"type": "fixed",
"value": "Standard_DS3_v2",
"hidden": true
},
"num_workers": {
"type": "fixed",
"value": 0,
"hidden": true
}
}

Note: For Lakeflow Spark Declarative Pipelines (SDP) policies, you must include "cluster_type" set to "dlt" in the policy definition.

Setting num_workers to 0 gives you a single-node (driver-only) compute resource, which is the most quota-friendly option for an Azure Free Trial.

RECOMMENDED STEPS FOR YOUR SITUATION

1. Delete the existing pipeline that is stuck with the wrong configuration.

2. Create a cluster policy (if you have not already) with fixed small VM types and cluster_type set to "dlt". You can verify your available VM sizes in the Azure portal to find one that fits within your quota.

3. Recreate the pipeline using the CLI command above, referencing your policy_id and setting apply_policy_default_values to true.

4. If you want to be explicit and not rely on the policy alone, also specify node_type_id and num_workers: 0 in the clusters block of the pipeline JSON.

RELEVANT DOCUMENTATION

- Configure classic compute for Lakeflow Spark Declarative Pipelines: https://docs.databricks.com/aws/en/delta-live-tables/configure-compute.html
- Pipeline settings reference: https://docs.databricks.com/aws/en/delta-live-tables/settings.html
- Compute policy definition reference: https://docs.databricks.com/aws/en/admin/clusters/policy-definition.html
- Pipelines CLI reference: https://docs.databricks.com/aws/en/dev-tools/cli/databricks-cli.html

Since you are on Azure, replace "aws" in the documentation URLs above with "azure" to see the Azure-specific versions.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.