topic Re: Cluster logs folder in Get Started Discussions

Cluster logs folder

fridthoy — Thu, 24 Oct 2024 10:13:54 GMT

Hi,

I can't see to find the cluster_logs folder, anyone that can help me find where the cluster logs are stored?

Best regards

Re: Cluster logs folder

gchandra — Thu, 24 Oct 2024 11:06:33 GMT

Check your cluster, where its pointing to.

Re: Cluster logs folder

fridthoy — Thu, 24 Oct 2024 11:37:29 GMT

It is pointing to none destination, but i can still see logs when i open my cluster

Re: Cluster logs folder

gchandra — Thu, 24 Oct 2024 11:58:33 GMT

If you want to save it as a files, please choose default dbfs:/cluster-logs here.

Re: Cluster logs folder

fridthoy — Thu, 24 Oct 2024 14:00:18 GMT

will cluster logs still be saved even if i have not specified a location? My storage cost has increased a lot without extra data in the storage, i think it is because of the logs

Re: Cluster logs folder

filipniziol — Thu, 24 Oct 2024 18:02:45 GMT

HI @fridthoy ,

If you experience high storage costs, it may be worth checking if you run regularly VACUUM command on the existing delta tables.

How Does Not Performing VACUUM Affect Storage?

Delta Lake Architecture: Delta Lake maintains a transaction log (_delta_log) that tracks all changes to the tables. When you perform operations like UPDATE, DELETE, or MERGE, Delta Lake retains previous versions of data files to support features like time travel and versioning.

Accumulation of Old Files: Without regular maintenance, these old data files can accumulate, leading to increased storage usage. This is especially true for tables with frequent write operations.

In summary, I do not think cluster logs are the cause of the increased storage costs, as normally they are just a small fraction of your overall storage. I would start investigating the storage related to the existing tables that are undergoing regular refreshes.

Re: Cluster logs folder

fiff — Fri, 25 Oct 2024 07:03:14 GMT

Thank you for the help!

I have enabled predictive optimization for unity catalog, thinking it would automatically preform VACCUM on the tables i have in my delta lake. With that in mind, I assumed VACCUM wouldn't require further attention.

Would it be better to manually run VACCUM on my tables, or is predictive optimization sufficient?

Best regards

Re: Cluster logs folder

filipniziol — Fri, 25 Oct 2024 07:32:25 GMT

Hi @fiff ,

According to documentation predictive optimization is doing VACUUM automatically.
However, there are a couple of exceptions: