- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2024 04:08 AM
As primary/foreign key constraints are now supported/available in Databricks, how are foreign key constraints handled in a dlt pipeline, i.e if a foreign key constraint is violated, is the record logged as a data quality issue and still added to the target table?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2024 04:47 AM
Hi @Mario_D ,
I don't think primary/foreign constraint are supported by DLT. At least I can't find anything in documentation. But you obtain same result using DLT expectations:
Manage data quality with Delta Live Tables - Azure Databricks | Microsoft Learn
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-16-2024 09:52 AM
Hi @Mario_D!
While primary & foreign key constraints are generally available in Databricks Runtime 15.2 and above, they are strictly informational only.
This means that a primary key will not prevent duplicates from being added to a table and a foreign key does not require the referenced value to exist in another table.
