cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

ef-zee
New Contributor III

I am trying to execute a DLT pipeline, but I am getting an error which says -

"INVALID_PARAMETER_VALUE: The field 'node_type_id' cannot be supplied when an instance pool ID is provided."

I am using my company's Azure Databricks platform with premium benefits.

PS: I am using the already available clusters, which work fine with Databricks notebooks ( I do not have admin access).

This is the configuration I have given.:

{
 
  "id": "57ed0e65-b9c4-4773-bc88-7dfe84be1dd1",
 
  "clusters": [
 
    {
 
      "label": "default",
 
      "policy_id": "C961B9761B0003BE",
 
      "autoscale": {
 
        "min_workers": 1,
 
        "max_workers": 5,
 
        "mode": "LEGACY"
 
      }
 
    },
 
    {
 
      "label": "maintenance",
 
      "policy_id": "C961B9761B0003BE"
 
    }
 
  ],
 
  "development": true,
 
  "continuous": false,
 
  "channel": "PREVIEW",
 
  "edition": "CORE",
 
  "photon": false,
 
  "libraries": [
 
    {
 
      "notebook": {
 
        "path": "/Users/Data Quality- PROD"
 
      }
 
    }
 
  ],
 
  "name": "dlt_demo",
 
  "storage": "dbfs:/pipelines/57ed0e65-b9c4-4773-bc88-7dfe84be1dd1"
 
}

What I have tried so far:

  1. Selecting different product editions.
  2. Checking with both pipeline modes.
  3. Tried with different permutations of the channel, cluster mode, policy

Can someone please suggest how to resolve this issue?

3 REPLIES 3

Debayan
Esteemed Contributor III

Hi, the workaround seems to be is to use:

instance_pool_id directly in your pipeline settings.

"clusters": [

{

"label": "default",

"instance_pool_id": "0713-080933-line63-pool-r1jmrzte",

"num_workers": 5

}

],

ef-zee
New Contributor III

Hey, I tried this, but now I got a different error.

PERMISSION_DENIED: You need the 'attach' permission on instance pool 0713-080933-line63-pool-r1jmrzte to use this API.

Could you please help?

dlt_3

Debayan
Esteemed Contributor III

Do you have cluster ACL enabled?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group