Hi all,

I have a DLT pipeline as so:

raw -> cleansed (SCD2) -> curated.

'Raw' is utilizing autoloader, to continously read file from a datalake. These files can contain tons of duplicate, which causes our raw table to become quite large. Therefore, we would like to truncate raw periodically to save storage size.

I have added 'pipelines.reset.allowed': 'false' to our cleansed table to ensure we dont lose our historical changes, i have also tried adding the ignoreDeletes parameter to both raw and cleansed without any success.

How should i organize my pipeline if I want to be able to periodically truncating the raw table?