- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2022 06:04 AM
Is there a way to limit how many retries DLT Pipelines run when in "Production" mode? What is the key value pair I use in the configuration?
- Labels:
-
DLT Pipelines
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2022 07:08 AM
Hi @Kristian Foster,
Yes. We have two spark configuration that can be set. pipelines.numStreamRetryAttempts and pipelines.numUpdateRetryAttempts. The former configures how many times we retry each flow before failing the update. The latter configures how many times we retry the entire pipeline before marking it as a fatally failed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2022 07:13 AM
Thanks @Cedric Law Hing Ping , that is what I was looking for in docs. Is there a doc you can share that lists all the options? Just so I can bookmark for future reference?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2022 08:12 AM
Hi @Kristian Foster,
Glad to help you out! I haven't seen this anywhere in the docs either specifically regarding this. I'll ask the docs team. Meanwhile, please feel free to bookmark https://docs.databricks.com/workflows/delta-live-tables/index.html as it contains the latest information regarding DLT.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2022 08:15 AM
Thanks @Cedric Law Hing Ping , it would be nice if they could update the docs with that information. Thanks again.
