How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

[ef-zee]

I am trying to execute a DLT pipeline, but I am getting an error which says -

"INVALID_PARAMETER_VALUE: The field 'node_type_id' cannot be supplied when an instance pool ID is provided."

I am using my company's Azure Databricks platform with premium benefits.

PS: I am using the already available clusters, which work fine with Databricks notebooks ( I do not have admin access).

This is the configuration I have given.:

{
 
  "id": "57ed0e65-b9c4-4773-bc88-7dfe84be1dd1",
 
  "clusters": [
 
    {
 
      "label": "default",
 
      "policy_id": "C961B9761B0003BE",
 
      "autoscale": {
 
        "min_workers": 1,
 
        "max_workers": 5,
 
        "mode": "LEGACY"
 
      }
 
    },
 
    {
 
      "label": "maintenance",
 
      "policy_id": "C961B9761B0003BE"
 
    }
 
  ],
 
  "development": true,
 
  "continuous": false,
 
  "channel": "PREVIEW",
 
  "edition": "CORE",
 
  "photon": false,
 
  "libraries": [
 
    {
 
      "notebook": {
 
        "path": "/Users/Data Quality- PROD"
 
      }
 
    }
 
  ],
 
  "name": "dlt_demo",
 
  "storage": "dbfs:/pipelines/57ed0e65-b9c4-4773-bc88-7dfe84be1dd1"
 
}

What I have tried so far:

1. Selecting different product editions.
2. Checking with both pipeline modes.
3. Tried with different permutations of the channel, cluster mode, policy

Can someone please suggest how to resolve this issue?

[Debayan]

Hi, the workaround seems to be is to use:

instance_pool_id directly in your pipeline settings.

"clusters": [

{

"label": "default",

"instance_pool_id": "0713-080933-line63-pool-r1jmrzte",

"num_workers": 5

}

],

[ef-zee]

Hey, I tried this, but now I got a different error.

PERMISSION_DENIED: You need the 'attach' permission on instance pool 0713-080933-line63-pool-r1jmrzte to use this API.

Could you please help?

[Debayan]

Do you have cluster ACL enabled?