DLT Pipeline Error Handling

[dashawn]

Hello all.

We are a new team implementing DLT and have setup a number of tables in a pipeline loading from s3 with UC as the target. I'm noticing that if any of the 20 or so tables fail to load, the entire pipeline fails even when there are no dependencies between the tables. In our case, a new table was added to the DLT notebook but the source s3 directory is empty. This has caused the pipeline to fail with error "org.apache.spark.sql.catalyst.ExtendedAnalysisException: Unable to process statement for Table 'table_name'.

Is there a way to change this behavior in the pipeline configuration so that one table failing doesn't impact the rest of the pipeline?

[Kaniz]

Hi @dashawn, 

• When data processing fails, manual investigation of logs to understand the failures, data cleanup, and determining the restart point can be time-consuming and costly. DLT provides features to handle errors more intelligently.
• By default, if any table fails to load, the entire pipeline fails. However, you can customize this behaviour to allow other tables to continue processing even if one table encounters an error.

[jose_gonzalez]

Thank you for sharing this @Kaniz. @dashawn did you were able to check Kaniz's docs? do you still need help or shall you accept Kaniz's solution?