cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Writing Records Failing Constraint Requirements to Separate Table when using Delta Live Tables

SQLArchitect
New Contributor

Are there any plans / capabilities in place or approaches people are using for writing (logging) records failing constraint requirements to separate tables when using Delta Live Tables?

Also, are there any plans / capabilities in place or approaches people are using for capturing which constraints have been violated similarly?

1 REPLY 1

Ryan_Chynoweth
Esteemed Contributor

According to the language reference documentation, I do not believe quarantining records is possible right now out of the box. But there are a few workarounds under the current functionality.

  1. Create a second table with the inverse of the expectations/constraints. This means that when records pass the original expectations then they will be inserted as expected, but with the inverse expectations only the bad records will pass the test and be inserted into a second table where you can then review them.
  2. Create some sort of case statement to identify if records do not pass expectations and add a "flag" column for the row. This would have a single table containing both good and bad records.

I would recommend option 1, as it is more inline with what you are requesting. Plus if/when DLT does add first class support for quarantine then you will be set up to easily switch over.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group