unvalidated the primary and foreign keys constraints?

jeremy98 — Sun, 15 Dec 2024 16:57:06 GMT

Hello community,
I'm inserting in a table defined (with primary key and foreign key set) some records in overwrite mode every moment I run a workflow where the task is defined. Why after inserting those records the DDL schema changes? Why I have my primary and foreign keys unset? Seems that changes the schema!

Re: unvalidated the primary and foreign keys constraints?

Alberto_Umana — Sun, 15 Dec 2024 23:43:07 GMT

Hi @jeremy98,

When you use the "insert overwrite" mode in Databricks, it can lead to the schema being reset, which includes the removal of primary and foreign key constraints. This happens because the "insert overwrite" operation essentially replaces the entire table, which can result in the loss of any constraints that were previously defined

To maintain your primary and foreign key constraints, you might consider using a different strategy for updating your table. For example, using the "merge" strategy instead of "insert overwrite" can help preserve the schema and constraints. The "merge" strategy allows you to update, insert, or delete records based on a condition, which can be more efficient and less disruptive to the schema

topic unvalidated the primary and foreign keys constraints? in Data Engineering

unvalidated the primary and foreign keys constraints?

Re: unvalidated the primary and foreign keys constraints?