Databricks Community

fridthoy · 4 weeks ago

Hi,

I can't see to find the cluster_logs folder, anyone that can help me find where the cluster logs are stored?

Best regards

gchandra · 4 weeks ago

Check your cluster, where its pointing to.

~

fridthoy · 4 weeks ago

It is pointing to none destination, but i can still see logs when i open my cluster

gchandra · 4 weeks ago

If you want to save it as a files, please choose default dbfs:/cluster-logs here.

~

fridthoy · 4 weeks ago

will cluster logs still be saved even if i have not specified a location? My storage cost has increased a lot without extra data in the storage, i think it is because of the logs

filipniziol · 4 weeks ago

HI @fridthoy ,

If you experience high storage costs, it may be worth checking if you run regularly VACUUM command on the existing delta tables.

How Does Not Performing VACUUM Affect Storage?

Delta Lake Architecture: Delta Lake maintains a transaction log (_delta_log) that tracks all changes to the tables. When you perform operations like UPDATE, DELETE, or MERGE, Delta Lake retains previous versions of data files to support features like time travel and versioning.

Accumulation of Old Files: Without regular maintenance, these old data files can accumulate, leading to increased storage usage. This is especially true for tables with frequent write operations.

In summary, I do not think cluster logs are the cause of the increased storage costs, as normally they are just a small fraction of your overall storage. I would start investigating the storage related to the existing tables that are undergoing regular refreshes.

fiff · 3 weeks ago

Thank you for the help!

I have enabled predictive optimization for unity catalog, thinking it would automatically preform VACCUM on the tables i have in my delta lake. With that in mind, I assumed VACCUM wouldn't require further attention.

Would it be better to manually run VACCUM on my tables, or is predictive optimization sufficient?

Best regards