DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.

Shanmugaraja — Fri, 22 May 2026 13:30:05 GMT

In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.

I created an instance pool with Instance type: Standard_DS3_v2 used inside a DLT policy setting .

Used "instance_pool_id" and "driver_instance_pool_id" for the VM compute series selection and made "driver_node_type_id" & "node_type_id" as forbidden.

When I run a DLT pipeline with a cluster policy applied, the cluster that comes up is:

Using E8 or F8 series VMs
Instead of the expected Standard_DS3_v2 from the pool.
On Job compute UI - its shows as Preferred Compute as E8/F8.

My question is

Does DLT ignore instance pool instance types ?
What if both are present - It still picks up E8 or F8 series VMs

Thanks in Advance

Sample compute policy for DLT

{ "cluster_type": { "type": "fixed", "value": "dlt" }, "spark_conf.spark.databricks.cluster.profile": { "type": "forbidden", "hidden": true }, "spark_version": { "type": "unlimited", "defaultValue": "auto:latest-lts" }, "node_type_id": { "type": "unlimited", "defaultValue": "Standard_DS3_v2", "isOptional": true }, "num_workers": { "type": "unlimited", "defaultValue": 1, "isOptional": true }, "azure_attributes.availability": { "type": "unlimited", "defaultValue": "SPOT_WITH_FALLBACK_AZURE" }, "azure_attributes.spot_bid_max_price": { "type": "fixed", "value": 100, "hidden": true }, "instance_pool_id": { "type": "allowlist", "values": [ "1111-100620-hurls114-pool-12345678", "1111-100620-vegan115-pool-12345678" ], "defaultValue": "1111-100620-hurls114-pool-12345678", "isOptional": true, "hidden": false }, "driver_instance_pool_id": { "type": "allowlist", "values": [ "1111-100620-hurls114-pool-12345678", "1111-100620-vegan115-pool-12345678" ], "defaultValue": "1111-100620-hurls114-pool-12345678", "isOptional": true, "hidden": false }, "autoscale.min_workers": { "type": "unlimited", "defaultValue": 1, "isOptional": true }, "autoscale.max_workers": { "type": "unlimited", "defaultValue": 4, "isOptional": true }, "runtime_engine": { "type": "unlimited", "defaultValue": "STANDARD" } }

topic DLT pipeline's compute policy when Instance pool Id used it ignores the VM series. in Data Engineering

DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.