Re: Error trying to edit Job Cluster via Databrick...

Louis_Frolio · ‎10-29-2025

@Adam_Borlase , Thanks, this is helpful context. The key is that the SQL Server connector’s ingestion pipeline runs on serverless, while the ingestion “gateway” runs on classic compute in your cloud account, so vCPU family quotas can block gateway creation unless you control instance types at creation time with a compute policy and/or the API.

What’s happening and why quotas matter

The ingestion gateway runs continuously on classic compute, inside your VNet/VPC, to capture snapshots and change logs; this consumes your cloud vCPUs and can be blocked by per‑VM‑family quotas. You must size/control its VM family during creation.
The ingestion pipeline (that applies staged changes into streaming tables) runs on serverless compute and does not consume your cloud vCPU quotas. You don’t need to control instance families there; you can only select serverless performance mode and apply serverless budget policies for tagging.
Applying a custom policy for the gateway is currently API‑only (UI doesn’t expose policy selection for the gateway yet). That’s why using the wizard left you without an option to change compute and resulted in quota errors when the default driver VM family wasn’t permitted.

Best-practice setup to enforce your compute policy at creation

Use either Databricks Asset Bundles (DAB) or the Pipelines API/CLI to create the gateway and ingestion pipeline while attaching a policy that locks the gateway instance family to one you have quota for.

1) Create a gateway‑specific compute policy

Define a dlt policy that fixes allowed node types and enforces UC‑compatible access mode. Databricks recommends using the smallest worker nodes for the gateway because they don’t affect performance; the minimum requirement is 8 cores on the driver to extract changes efficiently.

Example policy definition (adjust instance types to the VM family you have quota for): ```json { "cluster_type": { "type": "fixed", "value": "dlt" },

"data_security_mode": { "type": "fixed", "value": "USER_ISOLATION" },

"driver_node_type_id": { "type": "allowlist", "values": ["Standard_D8s_v5", "Standard_E8d_v4"], "defaultValue": "Standard_D8s_v5" },

"node_type_id": { "type": "allowlist", "values": ["Standard_D8s_v5", "Standard_E8d_v4"], "defaultValue": "Standard_D8s_v5" },

"num_workers": { "type": "fixed", "value": 1, "hidden": true } } ```

Notes: * Many workspaces default the gateway driver to a VM in the EDv4 family (for example Standard_E8d_v4); if that family is quota‑blocked in your region, pick another family that you know has capacity (e.g., Dv5) and put it in the policy allowlist/default.

Grant yourself permission on the policy so the API can attach it during pipeline creation.

2) Create the ingestion gateway with the policy attached

Create the gateway via CLI/API and include the policy on the clusters block; also apply policy defaults:

bash
databricks pipelines create --json '{
  "name": "sqlserver-gateway",
  "gateway_definition": {
    "connection_id": "<CONNECTION_ID>",
    "gateway_storage_catalog": "main",
    "gateway_storage_schema": "sqlserver01",
    "gateway_storage_name": "sqlserver01-gateway"
  },
  "clusters": [{
    "label": "default",
    "policy_id": "<POLICY_ID>",
    "apply_policy_default_values": true
  }]
}'

This ensures the gateway driver/worker types are enforced by your policy at creation time, preventing Databricks from picking an instance family you don’t have quota for.

3) Create the ingestion pipeline (serverless)

Create the ingestion pipeline pointing at the gateway; it runs on serverless automatically, so you don’t need quotas or node type control there:

bash
databricks pipelines create --json '{
  "name": "sqlserver-ingestion-pipeline",
  "ingestion_definition": {
    "ingestion_gateway_id": "<GATEWAY_PIPELINE_ID>",
    "objects": [
      { "schema": {
          "source_catalog": "sqlserver01",
          "source_schema": "dbo",
          "destination_catalog": "main",
          "destination_schema": "sqlserver01"
      }}
    ]
  }
}'

Optionally, if your account requires serverless budget policies for tagging, select one in the UI when you create/edit the pipeline; this only affects tagging and not compute sizing.

### Common failure causes when updating the pipeline definition * Trying to attach the policy in the UI wizard for the gateway—currently policy attachment is API‑only for the gateway; use the pipelines create API/CLI with the clusters block as shown above.

Using a policy with disallowed/legacy access modes (for example, NO_ISOLATION); use UC‑compatible Standard (USER_ISOLATION) or Dedicated (SINGLE_USER) in your policy’s data_security_mode.
Including autotermination_minutes in a policy applied to Lakeflow Declarative Pipelines compute; pipeline clusters auto‑shutdown and this setting causes errors—omit it in policies used for pipelines/gateways.
Passing legacy spark confs like spark.databricks.cluster.profile in policies; forbid them instead to avoid access mode conflicts.

Do you need to raise cloud quotas, or can you control compute type?

You can avoid quota issues by enforcing the driver_node_type_id/node_type_id to a VM family with available quota using a compute policy and attaching it at creation time via the API, as shown above.
If even the smallest supported families don’t have quota in your region, you’ll need your infra team to request a quota increase on that VM family from the cloud provider; Databricks recommends validating cloud service quotas as part of capacity planning for Lakeflow Connect gateways.
The ingestion pipeline itself uses serverless compute, which doesn’t require VM family quotas in your subscription; only the gateway is affected by quotas today.

Optional: DABs (Asset Bundles) for repeatable deployments

You can also package the gateway and ingestion pipeline definitions into a bundle and deploy across dev/stage/prod; if you go this route, ensure your gateway resource includes the clusters/policy_id block so enforcement happens at deploy time.

Quick checklist

Write/assign a dlt compute policy that allowlists only instance families you have quota for and sets data_security_mode to USER_ISOLATION (Standard).
Create the gateway via API/CLI with the policy attached on clusters.label="default" and apply_policy_default_values=true.
Create the ingestion pipeline (serverless) pointing at the gateway; no quotas required.
If you still hit quota errors, have infra request a vCPU quota increase for the targeted VM family or switch to a family with quota in the policy allowlist.

Hope this helps, Louis.

View solution in original post