topic Re: Saving failed records with failed expectation name(s) in Data Engineering

Saving failed records with failed expectation name(s)

youcanlearn — Sun, 14 Jul 2024 12:46:05 GMT

Hi all,

I am using Databricks expectations to manage my data quality. But I wanted to save the failed records along side with the expectation name(s) - one or many - that the record failed. The only way I figure out is, not to use Databricks expectation but to build new column for each data quality rule.

Does anyone know of a way to do so with expectations?

Re: Saving failed records with failed expectation name(s)

iakshaykr — Sun, 14 Jul 2024 13:00:42 GMT

@youcanlearn Have you explore this : https://docs.databricks.com/en/delta-live-tables/expectations.html

Re: Saving failed records with failed expectation name(s)

youcanlearn — Wed, 17 Jul 2024 04:12:43 GMT

@iakshaykr yes, but there isnt any concrete examples of implementing it.