Is there a way to let the DLT pipeline retry by itself?

guangyi — Fri, 26 Jul 2024 06:07:41 GMT

I know I can make the workflow job retry automatically by adding following properties in the YAML file: max_retries or min_retry_interval_millis.

However, I cannot find similar attributes in any DLT pipeline document. When I ask copilot it gives this answer:

I try to add these two attributes in the asset bundle YAML file. After deploying on the prod, I found these two properties actually added on the DLT pipeline setting > advanced > configuration section, but it still cannot retry automatically

Is there a way to let the DLT retry by itself?

Re: Is there a way to let the DLT pipeline retry by itself?

szymon_dybczak — Fri, 26 Jul 2024 07:10:26 GMT

Hi @guangyi ,

In DLT you have following two properties that you can set:

pipelines.maxFlowRetryAttempts

Type: int

The maximum number of attempts to retry a flow before failing a pipeline update when a retryable failure occurs.

The default value is two. By default, when a retryable failure occurs, the Delta Live Tables runtime attempts to run the flow three times including the original attempt.

pipelines.numUpdateRetryAttempts

Type: int

The maximum number of attempts to retry an update before failing the update when a retryable failure occurs. The retry is run as a full update.

The default is five. This parameter applies only to triggered updates run in production mode. There is no retry when your pipeline runs in development mode.

You can also take a look on below answers:

- Delta Live Tables - RETRY_ON_FAILURE - Databricks Community - 5926

- Solved: DLT Pipeline Retries - Databricks Community - 30388

topic Re: Is there a way to let the DLT pipeline retry by itself? in Data Engineering

Is there a way to let the DLT pipeline retry by itself?

Re: Is there a way to let the DLT pipeline retry by itself?