**Subject:** Azure Databricks Pipeline Not Applying Cluster Policy and Failing with VM Quota Error

**Summary of Issue**

I am attempting to create and run a pipeline in Azure Databricks using a custom cluster policy to restrict compute resources due to Azure Free Trial subscription limits. However, the pipeline is not honoring the configured cluster policy and instead defaults to a larger compute configuration, which results in VM quota errors during cluster launch.

**Details**

* Environment: Azure Databricks (Free Trial Subscription on Azure)
* Objective: Create a pipeline with restricted compute using a custom cluster policy.
* Policy Configuration: A cluster policy was created to enforce smaller VM sizes and limited worker scaling.
* Pipeline Creation: The pipeline was created after defining the cluster policy.

**Observed Behavior**

1. The pipeline does not apply the configured cluster policy during creation.
2. Instead, the pipeline defaults to a compute configuration similar to:

* Worker type: Standard_F4s
* Autoscaling: Min workers = 1, Max workers = 5
3. Because the Azure Free Trial subscription has limited CPU quota, the cluster fails to start with the following error:

```
Error class: CLUSTER_LAUNCH_CLIENT_ERROR
SQL state: KDL01

Encountered Quota Exhaustion issue in your account:
The VM size you are specifying is not available.
QuotaExceeded: Operation could not be completed due to insufficient CPU quota.
```

4. Even after deleting the pipeline and recreating it, the cluster policy is still not being applied.
5. When attempting to update the pipeline configuration via CLI, another error occurs:

```
Modifying following parameter gateway_definition in pipeline settings is not allowed
```

This prevents modifying the pipeline compute configuration once it is created.

**Impact**

* The pipeline cannot start due to VM quota limitations.
* The custom cluster policy intended to enforce smaller compute resources is not being applied.
* Pipeline configuration cannot be updated due to the gateway_definition restriction.

**Expected Behavior**

* The pipeline should respect the configured cluster policy during creation.
* Compute configuration should align with the policy limits (smaller VM and restricted worker scaling).
* The pipeline cluster should launch successfully within the available Azure quota.

**Request**

Assistance is requested to determine:

1. Why the pipeline is not applying the configured cluster policy.
2. How to ensure pipelines respect cluster policy constraints during creation.
3. Recommended configuration for running pipelines within Azure Free Trial compute quotas.