DLT pipeline's compute policy when Instance pool Id used it ignores the VM series.

Shanmugaraja
New Contributor

‎05-22-2026 06:30 AM

Hi 

In Lake flow Spark Declarative Pipelines (formerly DLT) I’m trying to understand how instance pool, cluster policy and DLT pipeline interact, especially around instance type selection.

I created an instance pool with Instance type: Standard_DS3_v2  used inside a DLT policy setting . 

Used  "instance_pool_id" and "driver_instance_pool_id" for the VM compute series selection and made "driver_node_type_id" & "node_type_id" as forbidden. 

When I run a DLT pipeline with a cluster policy applied, the cluster that comes up is:

  • Using E8 or F8 series VMs
  • Instead of the expected Standard_DS3_v2 from the pool. 
  • On Job compute UI - its shows as Preferred Compute as E8/F8. 
My question is 
  • Does DLT ignore instance pool instance types ? 
  • What if both are present - It still picks up E8 or F8 series VMs
 Thanks in Advance
 
Sample compute policy for DLT
 
{
  "cluster_type": {
    "type": "fixed",
    "value": "dlt"
  },
  "spark_conf.spark.databricks.cluster.profile": {
    "type": "forbidden",
    "hidden": true
  },
  "spark_version": {
    "type": "unlimited",
    "defaultValue": "auto:latest-lts"
  },
  "node_type_id": {
    "type": "unlimited",
    "defaultValue": "Standard_DS3_v2",
    "isOptional": true
  },
  "num_workers": {
    "type": "unlimited",
    "defaultValue": 1,
    "isOptional": true
  },
  "azure_attributes.availability": {
    "type": "unlimited",
    "defaultValue": "SPOT_WITH_FALLBACK_AZURE"
  },
  "azure_attributes.spot_bid_max_price": {
    "type": "fixed",
    "value": 100,
    "hidden": true
  },
  "instance_pool_id": {
    "type": "allowlist",
    "values": [
      "1111-100620-hurls114-pool-12345678",
      "1111-100620-vegan115-pool-12345678"
    ],
    "defaultValue": "1111-100620-hurls114-pool-12345678",
    "isOptional": true,
    "hidden": false
  },
  "driver_instance_pool_id": {
    "type": "allowlist",
    "values": [
      "1111-100620-hurls114-pool-12345678",
      "1111-100620-vegan115-pool-12345678"
    ],
    "defaultValue": "1111-100620-hurls114-pool-12345678",
    "isOptional": true,
    "hidden": false
  },
  "autoscale.min_workers": {
    "type": "unlimited",
    "defaultValue": 1,
    "isOptional": true
  },
  "autoscale.max_workers": {
    "type": "unlimited",
    "defaultValue": 4,
    "isOptional": true
  },
  "runtime_engine": {
    "type": "unlimited",
    "defaultValue": "STANDARD"
  }
}​
 
0 Kudos

MoJaMa
Databricks Employee
Databricks Employee

3 weeks ago

Can you also share the pipeline spec/JSON? Feel free to redact anything sensitive.

0 Kudos

MoJaMa
Databricks Employee
Databricks Employee

3 weeks ago

I tried to reproduce and it worked as expected for me. Check your pipeline JSON to make sure your "clusters" spec there is explicitly mentioning the pool details.

Example:

  "pipeline_type": "WORKSPACE",
  "name": "mojama-dlt-classic-demo",
  "clusters": [
    {
      "label": "default",
      "instance_pool_id": "0602-190328-throe6-pool-guogi5jw",
      "policy_id": "000AA2BE733A7B2E",
      "driver_instance_pool_id": "0602-190328-throe6-pool-guogi5jw",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 2,
        "mode": "ENHANCED"
      }
    }

 

0 Kudos