Databricks Community

dalcuovidiu

Hello,I have a question related to APPLY AS DELETE WHEN...If the source table does not have a column that specifies whether a record was deleted, I am currently using a workaround by ingesting synthetic data with a soft_deletion flag. In the future, ...

dalcuovidiu

Yes, I know. I also had a request in the past to delete records older than 5 years due to GDPR rules, so we created a procedure that automatically deletes them based on a threshold. Especially in Europe, this is a common issue. On-prem was pretty eas...

dalcuovidiu

I'm not entirely sure if I’m missing something here, but as far as I know there’s a golden rule in DWH applications: you never hard delete records, you use soft deletes instead. So I’m a bit puzzled why a hard delete is being used in this case.

dalcuovidiu

In sql, with a subselect works just fine %sql COPY INTO tabele_copy_into_test from ( SELECT user_id, email, _metadata.file_name AS source_filefrom "/Volumes/dbacademy_ecommerce/v01/raw/users-historical/")FILEFORMAT = PARQUETCOPY_OPTIONS ('m...

dalcuovidiu

Yep, I was expecting something similar to MERGE to avoid this kind of workaround. Maybe it will be released in the future.

dalcuovidiu

Ok, I will give it a try. I was reading the documentation , and it’s not very clear about the most common use case—when there isn’t a dedicated column that indicates a record was deleted in the source system. In my case, I just created such a column ...

Databricks Community

User Stats

User Activity

DLT - SCD 2 - detect deletes

Re: Rollbacks/deletes on streaming table

Re: Rollbacks/deletes on streaming table

Re: Reference file name when using COPY INTO?

Re: DLT - SCD 2 - detect deletes

Re: DLT - SCD 2 - detect deletes