https://community.databricks.com/t5/data-engineering/handling-gdpr-requests-in-databricks/m-p/42571#M27407 <P>When dealing with GDPR requests in <STRONG>databricks</STRONG>, there are some essential things to keep in mind:</P><P>- Use a low retention period to ensure you don't keep table delta version history for tables with personal information.</P><P>- Use APPLY CHANGES to handle Slowly Changing Dimension type 1. This way, you won't track history (like in type 2) and will have it in a separate table.</P><P>- When handling customer insertion and GDPR requests, use a changed data feed in databricks. Ensure the table is declared as LIVE, not STREAM, to ensure complete data reload and avoid records for which we have received GDPR requests.</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ezgif-3-020e69a4fd.gif" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/3289iB30A84A4506D97CB/image-size/large?v=v2&amp;px=999" role="button" title="ezgif-3-020e69a4fd.gif" alt="ezgif-3-020e69a4fd.gif" /></span></P> Mon, 28 Aug 2023 17:01:37 GMT Hubert-Dudek 2023-08-28T17:01:37Z https://community.databricks.com/t5/data-engineering/handling-gdpr-requests-in-databricks/m-p/42571#M27407 <P>When dealing with GDPR requests in <STRONG>databricks</STRONG>, there are some essential things to keep in mind:</P><P>- Use a low retention period to ensure you don't keep table delta version history for tables with personal information.</P><P>- Use APPLY CHANGES to handle Slowly Changing Dimension type 1. This way, you won't track history (like in type 2) and will have it in a separate table.</P><P>- When handling customer insertion and GDPR requests, use a changed data feed in databricks. Ensure the table is declared as LIVE, not STREAM, to ensure complete data reload and avoid records for which we have received GDPR requests.</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="ezgif-3-020e69a4fd.gif" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/3289iB30A84A4506D97CB/image-size/large?v=v2&amp;px=999" role="button" title="ezgif-3-020e69a4fd.gif" alt="ezgif-3-020e69a4fd.gif" /></span></P> Mon, 28 Aug 2023 17:01:37 GMT https://community.databricks.com/t5/data-engineering/handling-gdpr-requests-in-databricks/m-p/42571#M27407 Hubert-Dudek 2023-08-28T17:01:37Z https://community.databricks.com/t5/data-engineering/handling-gdpr-requests-in-databricks/m-p/42894#M27443 <P>Thank you for sharing this information&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/25346">@Hubert-Dudek</a>!!!!</P> Wed, 30 Aug 2023 17:26:29 GMT https://community.databricks.com/t5/data-engineering/handling-gdpr-requests-in-databricks/m-p/42894#M27443 jose_gonzalez 2023-08-30T17:26:29Z