topic Re: How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline? in Data Engineering

How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

ef-zee — Wed, 28 Sep 2022 09:59:30 GMT

I am trying to execute a DLT pipeline, but I am getting an error which says -

"INVALID_PARAMETER_VALUE: The field 'node_type_id' cannot be supplied when an instance pool ID is provided."

I am using my company's Azure Databricks platform with premium benefits.

PS: I am using the already available clusters, which work fine with Databricks notebooks ( I do not have admin access).

This is the configuration I have given.:

{
 
  "id": "57ed0e65-b9c4-4773-bc88-7dfe84be1dd1",
 
  "clusters": [
 
    {
 
      "label": "default",
 
      "policy_id": "C961B9761B0003BE",
 
      "autoscale": {
 
        "min_workers": 1,
 
        "max_workers": 5,
 
        "mode": "LEGACY"
 
      }
 
    },
 
    {
 
      "label": "maintenance",
 
      "policy_id": "C961B9761B0003BE"
 
    }
 
  ],
 
  "development": true,
 
  "continuous": false,
 
  "channel": "PREVIEW",
 
  "edition": "CORE",
 
  "photon": false,
 
  "libraries": [
 
    {
 
      "notebook": {
 
        "path": "/Users/Data Quality- PROD"
 
      }
 
    }
 
  ],
 
  "name": "dlt_demo",
 
  "storage": "dbfs:/pipelines/57ed0e65-b9c4-4773-bc88-7dfe84be1dd1"
 
}

What I have tried so far:

Selecting different product editions.
Checking with both pipeline modes.
Tried with different permutations of the channel, cluster mode, policy

Can someone please suggest how to resolve this issue?

Re: How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

ef-zee — Wed, 28 Sep 2022 15:42:01 GMT

Hey, I tried this, but now I got a different error.

PERMISSION_DENIED: You need the 'attach' permission on instance pool 0713-080933-line63-pool-r1jmrzte to use this API.

Could you please help?

Re: How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

Debayan — Wed, 28 Sep 2022 13:45:13 GMT

Hi, the workaround seems to be is to use:

instance_pool_id directly in your pipeline settings.

"clusters": [

{

"label": "default",

"instance_pool_id": "0713-080933-line63-pool-r1jmrzte",

"num_workers": 5

}

Re: How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?

Debayan — Thu, 29 Sep 2022 13:39:02 GMT

Do you have cluster ACL enabled?