topic Re: How to enforce a cleanup policy on job cluster logs in Data Engineering

How to enforce a cleanup policy on job cluster logs

EDDatabricks — Wed, 13 Sep 2023 14:07:09 GMT

We have a number of jobs on our databricks workspaces. All job clusters are configured with a dbfs location to save the respective logs (configured from Job cluster -> "Advanced options" -> "Logging").

However, the logs are retained in the dbfs indefinitely, even after 60 days (which is the retention period for job runs history). Is it possible to configure a clean up policy on logs older than a defined threshold via the databricks workspace? If it's not possible to achieve this within databricks, what would be a best practice to do this on Azure?

Re: How to enforce a cleanup policy on job cluster logs

Atanu — Wed, 13 Sep 2023 15:26:30 GMT

@EDDatabricks Thanks for reaching out to us.

I think you should explore our Purge option is enabled or not.
https://docs.databricks.com/en/administration-guide/workspace/settings/storage.html

Re: How to enforce a cleanup policy on job cluster logs

mtz629 — Thu, 10 Oct 2024 17:52:32 GMT

Pretty sure that controls what's available to the UI in terms of cluster logs, not what resides in the cluster-logs storage directory. Those are two different considerations.