cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Writing Records Failing Constraint Requirements to Separate Table when using Delta Live Tables

SQLArchitect
New Contributor

Are there any plans / capabilities in place or approaches people are using for writing (logging) records failing constraint requirements to separate tables when using Delta Live Tables?

Also, are there any plans / capabilities in place or approaches people are using for capturing which constraints have been violated similarly?

1 REPLY 1

Ryan_Chynoweth
Honored Contributor III

According to the language reference documentation, I do not believe quarantining records is possible right now out of the box. But there are a few workarounds under the current functionality.

  1. Create a second table with the inverse of the expectations/constraints. This means that when records pass the original expectations then they will be inserted as expected, but with the inverse expectations only the bad records will pass the test and be inserted into a second table where you can then review them.
  2. Create some sort of case statement to identify if records do not pass expectations and add a "flag" column for the row. This would have a single table containing both good and bad records.

I would recommend option 1, as it is more inline with what you are requesting. Plus if/when DLT does add first class support for quarantine then you will be set up to easily switch over.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.