Are there any ways to automatically cleanup temporary files created in s3 by the Amazon Redshift connector

[sajith_appukutt]

The Amazon Redshift data source in Databricks seems to be using S3 for storing intermediate results. Are there any ways to automatically cleanup temporary files created in S3

[sajith_appukutt · Accepted solution]

You could use storage lifecycle policy for the s3 bucket used for storing intermediate results and configure expiration actions. This way temporary/intermediate results would be automatically cleaned up