HI @fridthoy ,
If you experience high storage costs, it may be worth checking if you run regularly VACUUM command on the existing delta tables.
How Does Not Performing VACUUM Affect Storage?
Delta Lake Architecture: Delta Lake maintains a transaction log (_delta_log) that tracks all changes to the tables. When you perform operations like UPDATE, DELETE, or MERGE, Delta Lake retains previous versions of data files to support features like time travel and versioning.
Accumulation of Old Files: Without regular maintenance, these old data files can accumulate, leading to increased storage usage. This is especially true for tables with frequent write operations.
In summary, I do not think cluster logs are the cause of the increased storage costs, as normally they are just a small fraction of your overall storage. I would start investigating the storage related to the existing tables that are undergoing regular refreshes.
