topic Re: Getting in Data Engineering

Getting

anantkharat — Thu, 28 Nov 2024 10:39:09 GMT

payload = {

"clusters": [

{

"num_workers": 4

}

"pipeline_id": pipeline_id

}

update_url = f"{workspace_url}/api/2.0/pipelines/{pipeline_id}"

response = requests.put(update_url, headers=headers, json=payload)

for this, i'm getting below output with status code 400.
{"error_code":"INVALID_PARAMETER_VALUE","message":"Cannot modify catalog of an existing pipeline. Existing catalog: 'doha'. Requested catalog: ''"}

i also have tried with

url = f"{workspace_url}/api/2.0/pipelines/{pipeline_id}"

response = requests.get(url, headers=headers)

response.json()

using above response and modifying the num_workers, without changing any catalog parameter and also with catalog parameter same as previous. but still getting same error..

kindly help

Re: Getting

NandiniN — Mon, 02 Dec 2024 11:28:18 GMT

Hi @anantkharat ,

Can you please try something like
pipelines.update(pipeline_id, configuration=configuration, catalog='doha')

From the error message it says catalog was empty.

Check this section https://docs.databricks.com/api/workspace/pipelines/update#catalog

A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

Thanks!

Re: Getting

NandiniN — Wed, 04 Dec 2024 09:34:09 GMT

Thank you for accepting the solution, I am glad it worked.