DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.

[Shanmugaraja]

Hi 

In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.

I created an instance pool with Instance type: Standard_DS3_v2  used inside a DLT policy setting . 

Used  "instance_pool_id" and "driver_instance_pool_id" for the VM compute series selection and made "driver_node_type_id" & "node_type_id" as forbidden. 

When I run a DLT pipeline with a cluster policy applied, the cluster that comes up is:

• Using E8 or F8 series VMs
• Instead of the expected Standard_DS3_v2 from the pool. 
• On Job compute UI - its shows as Preferred Compute as E8/F8. 

My question is 

• Does DLT ignore instance pool instance types ? 
• What if both are present - It still picks up E8 or F8 series VMs

 Thanks in Advance

Sample compute policy for DLT

{
  "cluster_type": {
    "type": "fixed",
    "value": "dlt"
  },
  "spark_conf.spark.databricks.cluster.profile": {
    "type": "forbidden",
    "hidden": true
  },
  "spark_version": {
    "type": "unlimited",
    "defaultValue": "auto:latest-lts"
  },
  "node_type_id": {
    "type": "unlimited",
    "defaultValue": "Standard_DS3_v2",
    "isOptional": true
  },
  "num_workers": {
    "type": "unlimited",
    "defaultValue": 1,
    "isOptional": true
  },
  "azure_attributes.availability": {
    "type": "unlimited",
    "defaultValue": "SPOT_WITH_FALLBACK_AZURE"
  },
  "azure_attributes.spot_bid_max_price": {
    "type": "fixed",
    "value": 100,
    "hidden": true
  },
  "instance_pool_id": {
    "type": "allowlist",
    "values": [
      "1111-100620-hurls114-pool-12345678",
      "1111-100620-vegan115-pool-12345678"
    ],
    "defaultValue": "1111-100620-hurls114-pool-12345678",
    "isOptional": true,
    "hidden": false
  },
  "driver_instance_pool_id": {
    "type": "allowlist",
    "values": [
      "1111-100620-hurls114-pool-12345678",
      "1111-100620-vegan115-pool-12345678"
    ],
    "defaultValue": "1111-100620-hurls114-pool-12345678",
    "isOptional": true,
    "hidden": false
  },
  "autoscale.min_workers": {
    "type": "unlimited",
    "defaultValue": 1,
    "isOptional": true
  },
  "autoscale.max_workers": {
    "type": "unlimited",
    "defaultValue": 4,
    "isOptional": true
  },
  "runtime_engine": {
    "type": "unlimited",
    "defaultValue": "STANDARD"
  }
}​