Is there a way to let the DLT pipeline retry by itself?

[guangyi]

I know I can make the workflow job retry automatically by adding following properties in the YAML file: max_retries or min_retry_interval_millis.

However, I cannot find similar attributes in any DLT pipeline document. When I ask copilot it gives this answer:

I try to add these two attributes in the asset bundle YAML file. After deploying on the prod, I found these two properties actually added on the DLT pipeline setting > advanced > configuration section, but it still cannot retry automatically

Is there a way to let the DLT retry by itself?

[szymon_dybczak · Accepted solution]

Hi @guangyi ,

In DLT you have following two properties that you can set:

pipelines.maxFlowRetryAttempts Type: int The maximum number of attempts to retry a flow before failing a pipeline update when a retryable failure occurs. The default value is two. By default, when a retryable failure occurs, the Delta Live Tables runtime attempts to run the flow three times including the original attempt.

pipelines.numUpdateRetryAttempts Type: int The maximum number of attempts to retry an update before failing the update when a retryable failure occurs. The retry is run as a full update. The default is five. This parameter applies only to triggered updates run in production mode. There is no retry when your pipeline runs in development mode.

You can also take a look on below answers:

- Delta Live Tables - RETRY_ON_FAILURE - Databricks Community - 5926

- Solved: DLT Pipeline Retries - Databricks Community - 30388