Sanjeeb2024
Valued Contributor

Hi @RevanthV  - You can create a data validation module and before writing the data to target, you can first validate the data as part of your pipeline and then define a process how to handle bad data. You can explore great expectations library or soda as well. However if you put lots of validation it will also increase your ETL job pipeline execution.

 

Sanjeeb Mohapatra