Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I am implementing Structured Streaming using Delta Live Table. I want to delete the parquet files once they are used. What options should I set so that the files loaded in S3 are not deleted?
It sounds like your Structured Streaming source is S3, in which case the easiest solution is likely to manage the stream source using an S3 Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html). The objects could be managed by S3 Lifecycle via time-based trigger (e.g. 14 days) or it could be more granularly managed using tags such that after a file is processed, you call the AWS SDK, tag an object with a key/value pair such as `delete=true` and then the lifecycle cleans up objects with those tags.
Thanks.
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
