How can I configure Lakeflow Connect SQL Server CDC Gateway to use a desired VM type?

[Mado]

Hi Team,

I'm evaluating Databricks Lakeflow Connect for SQL Server CDC ingestion and have run into a gateway provisioning issue.

Environment

• Region: Australia East
• Source: Azure SQL Database
• CDC enabled successfully on the database and source table
• SQL Server connection created successfully in Lakeflow Connect

   

What I configured

I created a SQL Server ingestion pipeline using:

• Change Data Capture (CDC)
• Ingestion pipeline
• Ingestion gateway

  The SQL connection, gateway definition, and pipeline configuration were all created successfully.

   

Problem

When Lakeflow attempts to start the CDC gateway, the compute provisioning fails with the following error:

 

Message

Failed to add 2 workers to the compute. Will attempt retry: true. Reason:Cloud Provider Resource Stockout

Help

The VM size you are specifying is not available. To reduce future stockout errors, enable flexible node types if not already enabled so Databricks can automatically fall back to compatible instance types when capacity is limited. Note: flexible node types are not available for GPU instance types. [details] SkuNotAvailable: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_E4d_v4' is currently not available in location 'australiaeast'. Please try another size or deploy to a different location or different zone. 

Investigation

I inspected the generated compute configuration and found that Lakeflow is attempting to provision:

 

Driver: Standard_E4d_v4

 

Worker: Standard_D4s_v4

 

I also verified the following:

• Azure SQL connectivity works.
• CDC is functioning correctly.
• Flexible Node Types has been enabled on the workspace.

  The issue appears to be that Azure Australia East currently cannot allocate Standard_E4d_v4.

   

 

Question

Is there a supported way to configure or override the VM types used by a Lakeflow Connect SQL Server CDC Gateway?

Specifically:

1. Can the gateway be configured to use v5 VM families (for example Standard_E4ds_v5 and Standard_D4ds_v5) instead of the default v4 families?
2. Can this be configured during pipeline creation?
3. If not, what is the recommended approach when the default gateway VM SKU is unavailable in the selected Azure region?

   Any guidance would be greatly appreciated.

   

[ivanvyd]

@Mado this looks like a gateway compute capacity issue rather than a CDC issue.

Databricks does support customizing SQL Server gateway compute through a custom Job Compute policy/API. It is a good practice to keep workers small; requires at least 8 cores on the driver for efficient extraction.

Also, "Standard_E4d_v4" currently has no compatible Flexible Node Type fallback, so enabling Flexible Node Types won't help with this particular SKU.

I'd use an available v5 driver with 8+ cores through a custom policy/API, plus a small worker SKU available in Astralia East, rather than relying on the UI-generated defaults.

[ThomazNeto]

Hi Mado,

Ivan is right on the diagnosis. Adding the concrete "how", since the docs only give it in fragments.

1. Can the gateway use other VM families? Yes, but not from the wizard. The UI walkthrough has no compute step for the gateway; the docs list the policy route as "Unrestricted permissions to create clusters, or a custom policy (API only)". Two ways to set it:

• Cluster policy: family Job Compute, with the overrides the docs require ("cluster_type": fixed "dlt", "runtime_engine": fixed "STANDARD", "num_workers": unlimited, default 1) plus your node types, e.g. "driver_node_type_id": fixed "Standard_E8ds_v5" and "node_type_id": fixed "Standard_D4ds_v5". The docs' own Azure example is Standard_E64d_v4 driver and Standard_F4s workers, and they say workers "do not impact gateway performance", so keep them tiny.
• Directly in the gateway pipeline definition (bundles or Pipelines API): the gateway is a pipeline object and the docs show a commented clusters block on it, with label "default", autoscale, node_type_id and policy_id. That's where you pin the SKU without a policy.
  link

Two constraints to respect: the driver needs at least 8 cores ("The minimum requirement for the driver node is 8 cores"), so Standard_E4ds_v5 is too small for the driver, go E8ds_v5 or larger; and the gateway must be classic compute, serverless isn't an option for it.
link

1. During creation? Only via API, bundles or the Terraform examples repo linked from the docs. Editing the clusters block afterwards isn't documented for the gateway specifically, so I'd recreate the gateway from YAML for the POC rather than patch the UI-generated one.
2. Why Flexible Node Types didn't save you: Standard_E4d_v4 (the non-"s" variant) simply isn't in the compatibility table. Standard_E4ds_v4 falls back to E4ds_v5 and L4s, Standard_E4d_v4 has no entry, so there's nothing to fall back to. Pick the "ds" v5 SKUs, which are in the table and have their own fallbacks.
   link

Australia East stockouts on v4 E-series are a recurring thing, so pinning v5 through the policy is the right long-term fix, not a workaround.