cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Foreign key constraint in a dlt pipeline

Mario_D
New Contributor II

As primary/foreign key constraints are now supported/available in Databricks, how are foreign key constraints handled in a dlt pipeline, i.e if a foreign key constraint is violated, is the record logged as a data quality issue and still added to the target table?

1 ACCEPTED SOLUTION

Accepted Solutions

szymon_dybczak
Contributor III

Hi @Mario_D ,

I don't think primary/foreign constraint are supported by DLT. At least I can't find anything in documentation. But you obtain same result using DLT expectations:

Manage data quality with Delta Live Tables - Azure Databricks | Microsoft Learn

View solution in original post

2 REPLIES 2

szymon_dybczak
Contributor III

Hi @Mario_D ,

I don't think primary/foreign constraint are supported by DLT. At least I can't find anything in documentation. But you obtain same result using DLT expectations:

Manage data quality with Delta Live Tables - Azure Databricks | Microsoft Learn

RCo
New Contributor II

Hi @Mario_D!

While primary & foreign key constraints are generally available in Databricks Runtime 15.2 and above, they are strictly informational only.

This means that a primary key will not prevent duplicates from being added to a table and a foreign key does not require the referenced value to exist in another table.

https://docs.databricks.com/en/tables/constraints.html#declare-primary-key-and-foreign-key-relations...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group