Hi all,
We have the following use case and wondering if DLT is the correct approach.
Landing area with daily dumps of parquet files into our Data Lake container.
The daily dump does a full overwrite of the parquet each time, keeping the same file name.
The idea would be to re-process the whole parquet each time and manage the delta in the bronze table with SCD 2.
Suggestions on the best approach would be helpful.
Cheers
