topic Re: DLT Pipeline Retries in Data Engineering

DLT Pipeline Retries

kfoster — Wed, 28 Sep 2022 13:04:05 GMT

Is there a way to limit how many retries DLT Pipelines run when in "Production" mode? What is the key value pair I use in the configuration?

Re: DLT Pipeline Retries

Cedric — Wed, 28 Sep 2022 14:08:51 GMT

Hi @Kristian Foster,

Yes. We have two spark configuration that can be set. pipelines.numStreamRetryAttempts and pipelines.numUpdateRetryAttempts. The former configures how many times we retry each flow before failing the update. The latter configures how many times we retry the entire pipeline before marking it as a fatally failed.

Re: DLT Pipeline Retries

kfoster — Wed, 28 Sep 2022 14:13:14 GMT

Thanks @Cedric Law Hing Ping , that is what I was looking for in docs. Is there a doc you can share that lists all the options? Just so I can bookmark for future reference?

Re: DLT Pipeline Retries

Cedric — Wed, 28 Sep 2022 15:12:23 GMT

Hi @Kristian Foster,

Glad to help you out! I haven't seen this anywhere in the docs either specifically regarding this. I'll ask the docs team. Meanwhile, please feel free to bookmark https://docs.databricks.com/workflows/delta-live-tables/index.html as it contains the latest information regarding DLT.

Re: DLT Pipeline Retries

kfoster — Wed, 28 Sep 2022 15:15:01 GMT

Thanks @Cedric Law Hing Ping , it would be nice if they could update the docs with that information. Thanks again.