topic Re: Idempotency token in Data Engineering

Idempotency token

15460 — Fri, 08 Nov 2024 19:42:11 GMT

Hi Team,

I have used idempotency token in my dag code to avoid duplicate runs.
note: Idempotency token given as static value
Issue: If dag fails once ...because of this idempotency token, airflow is not allowing to connect dbx ...can you please help me with it.
Error:

airflow.exceptions.AirflowException: Response: b'{"error_code":"INVALID_PARAMETER_VALUE","message":"Retrieving the output of runs with multiple tasks is not supported. Please retrieve the output of each individual task run instead."}', Status Code: 400 [2024-11-07, 19:31:19 UTC] {{taskinstance.py:1278}} INFO - Marking task as FAILED. dag_id=S4_FACT_DL_INVENTORY_MOVEMENTS_RAW_CLEANSE, task_id=inventory_movements_silver_streaming, execution_date=20241107T192758, start_date=20241107T193118, end_date=20241107T193119 [2024-11-07, 19:31:19 UTC] {{standard_task_runner.py:93}} ERROR - Failed to execute job 267015 for task inventory_movements_silver_streaming (Response: b'{"error_code":"INVALID_PARAMETER_VALUE","message":"Retrieving the output of runs with multiple tasks is not supported. Please retrieve the output of each individual task run instead."}', Status Code: 400; 17)

I am not able to run the dag now. even though airflow job and dbx job both are not active. I am trying to run dag as new instance..still i am facing this issue. Kindly help me.

Re: Idempotency token

Vivian_Wilfred — Tue, 12 Nov 2024 17:44:02 GMT

Hi @15460 , are we sure this issue is related to Idempotency token? The error you are encountering may not be related to the Idempotency token. What is your airflow provider version? Can you please upgrade to the latest version and retry?

Re: Idempotency token

15460 — Tue, 12 Nov 2024 20:21:59 GMT

Hi Vivian,
Thanks for the response. Even i feel like it can be airflow issue. why because, even i dont have dbx job running at dbx end.. airflow still pointing to that idempotency token run id and results that error.
current version we are using : 2.2.5-python38

Re: Idempotency token

Vivian_Wilfred — Wed, 13 Nov 2024 00:18:03 GMT

I see some issues discussed in the airflow like this but it was fixed in the later versions. Try upgrading the provider version and let us know if it fixes the issue.