DLT Pipeline Retries

[kfoster]

Is there a way to limit how many retries DLT Pipelines run when in "Production" mode? What is the key value pair I use in the configuration?

[Cedric · Accepted solution]

Hi @Kristian Foster​,

Yes. We have two spark configuration that can be set. pipelines.numStreamRetryAttempts and pipelines.numUpdateRetryAttempts. The former configures how many times we retry each flow before failing the update. The latter configures how many times we retry the entire pipeline before marking it as a fatally failed.

[kfoster]

Thanks @Cedric Law Hing Ping​ , that is what I was looking for in docs. Is there a doc you can share that lists all the options? Just so I can bookmark for future reference?

[Cedric]

Hi @Kristian Foster​,

Glad to help you out! I haven't seen this anywhere in the docs either specifically regarding this. I'll ask the docs team. Meanwhile, please feel free to bookmark https://docs.databricks.com/workflows/delta-live-tables/index.html as it contains the latest information regarding DLT.

[kfoster]

Thanks @Cedric Law Hing Ping​ , it would be nice if they could update the docs with that information. Thanks again.