topic Re: Are there any ways to automatically cleanup temporary files created in s3 by the Amazon Redshift connector in Data Engineering https://community.databricks.com/t5/data-engineering/are-there-any-ways-to-automatically-cleanup-temporary-files/m-p/25475#M17715 <P>You could use <A href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html" alt="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html" target="_blank">storage lifecycle policy</A> for the s3 bucket used for storing intermediate results and configure <A href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html" alt="https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html" target="_blank">expiration actions</A>. This way temporary/intermediate results would be automatically cleaned up</P><P></P> Fri, 18 Jun 2021 00:29:49 GMT sajith_appukutt 2021-06-18T00:29:49Z Are there any ways to automatically cleanup temporary files created in s3 by the Amazon Redshift connector https://community.databricks.com/t5/data-engineering/are-there-any-ways-to-automatically-cleanup-temporary-files/m-p/25474#M17714 <P>The Amazon Redshift&nbsp;data source in Databricks seems to be using S3 for storing intermediate results. Are there any ways to automatically cleanup temporary files created in S3</P> Wed, 09 Jun 2021 05:22:38 GMT https://community.databricks.com/t5/data-engineering/are-there-any-ways-to-automatically-cleanup-temporary-files/m-p/25474#M17714 sajith_appukutt 2021-06-09T05:22:38Z Re: Are there any ways to automatically cleanup temporary files created in s3 by the Amazon Redshift connector https://community.databricks.com/t5/data-engineering/are-there-any-ways-to-automatically-cleanup-temporary-files/m-p/25475#M17715 <P>You could use <A href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html" alt="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html" target="_blank">storage lifecycle policy</A> for the s3 bucket used for storing intermediate results and configure <A href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html" alt="https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html" target="_blank">expiration actions</A>. This way temporary/intermediate results would be automatically cleaned up</P><P></P> Fri, 18 Jun 2021 00:29:49 GMT https://community.databricks.com/t5/data-engineering/are-there-any-ways-to-automatically-cleanup-temporary-files/m-p/25475#M17715 sajith_appukutt 2021-06-18T00:29:49Z