DLT Pipeline Error Handling
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-18-2024 07:54 PM
Hello all.
We are a new team implementing DLT and have setup a number of tables in a pipeline loading from s3 with UC as the target. I'm noticing that if any of the 20 or so tables fail to load, the entire pipeline fails even when there are no dependencies between the tables. In our case, a new table was added to the DLT notebook but the source s3 directory is empty. This has caused the pipeline to fail with error "org.apache.spark.sql.catalyst.ExtendedAnalysisException: Unable to process statement for Table 'table_name'.
Is there a way to change this behavior in the pipeline configuration so that one table failing doesn't impact the rest of the pipeline?
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2024 12:00 AM
@Retired_mod , could you please elaborate more on how to "allow other tables to continue processing even if one table encounters an error"?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2024 05:19 PM
Thank you for sharing this @Retired_mod. @dashawn did you were able to check Kaniz's docs? do you still need help or shall you accept Kaniz's solution?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-29-2024 12:21 PM
could please provide link for the docs

