https://community.databricks.com/t5/data-engineering/what-are-the-different-options-for-dealing-with-invalid-records/m-p/20954#M14201 <P>Delta Live Table supports the data quality checks via expectations. On encountering invalid records you can choose to either retain them, drop them or fail/stop the pipeline. See the link below for additional details</P><P></P><P><A href="https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables&amp;language-sql" target="test_blank">https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables&amp;language-sql</A></P> Thu, 24 Jun 2021 03:41:27 GMT aladda 2021-06-24T03:41:27Z https://community.databricks.com/t5/data-engineering/what-are-the-different-options-for-dealing-with-invalid-records/m-p/20953#M14200 Thu, 24 Jun 2021 03:40:03 GMT https://community.databricks.com/t5/data-engineering/what-are-the-different-options-for-dealing-with-invalid-records/m-p/20953#M14200 aladda 2021-06-24T03:40:03Z https://community.databricks.com/t5/data-engineering/what-are-the-different-options-for-dealing-with-invalid-records/m-p/20954#M14201 <P>Delta Live Table supports the data quality checks via expectations. On encountering invalid records you can choose to either retain them, drop them or fail/stop the pipeline. See the link below for additional details</P><P></P><P><A href="https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables&amp;language-sql" target="test_blank">https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#publish-tables&amp;language-sql</A></P> Thu, 24 Jun 2021 03:41:27 GMT https://community.databricks.com/t5/data-engineering/what-are-the-different-options-for-dealing-with-invalid-records/m-p/20954#M14201 aladda 2021-06-24T03:41:27Z