Can I delete specific partition from a Delta Live Table?

gauravchaturved — Wed, 22 May 2024 14:08:20 GMT

if I have created a Delta Live Table with partition on a column (lets say a date column) from a Stream Source, can I delete the partition for specific date values later to save on cost & to keep the table lean? if I can, then -

1- how to do it?

2- do I also need to run vacuum as second step?

Re: Can I delete specific partition from a Delta Live Table?

raphaelblg — Wed, 22 May 2024 22:52:30 GMT

Hello @gauravchaturved ,

You can remove the partition by filtering it in your source code and triggering a full refresh in your pipeline. There is no need to run vacuum, as DLT has maintenance clusters that perform OPTIMIZE and VACUUM operations on your DLT-defined tables.

Please note that DLT tables should not be modified outside of the DLT pipeline. Ensure that all necessary logic is included within your DLT pipeline.

topic Re: Can I delete specific partition from a Delta Live Table? in Data Engineering

Can I delete specific partition from a Delta Live Table?

Re: Can I delete specific partition from a Delta Live Table?