Handling files used more than once in a streaming pipeline

rt-slowth — Tue, 09 Jan 2024 23:57:52 GMT

I am implementing Structured Streaming using Delta Live Table. I want to delete the parquet files once they are used. What options should I set so that the files loaded in S3 are not deleted?

Re: Handling files used more than once in a streaming pipeline

brockb — Tue, 16 Jan 2024 02:24:06 GMT

Hi,

It sounds like your Structured Streaming source is S3, in which case the easiest solution is likely to manage the stream source using an S3 Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html). The objects could be managed by S3 Lifecycle via time-based trigger (e.g. 14 days) or it could be more granularly managed using tags such that after a file is processed, you call the AWS SDK, tag an object with a key/value pair such as `delete=true` and then the lifecycle cleans up objects with those tags.

Thanks.

topic Re: Handling files used more than once in a streaming pipeline in Data Engineering

Handling files used more than once in a streaming pipeline

Re: Handling files used more than once in a streaming pipeline