How to enforce a cleanup policy on job cluster logs
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2023 07:07 AM
We have a number of jobs on our databricks workspaces. All job clusters are configured with a dbfs location to save the respective logs (configured from Job cluster -> "Advanced options" -> "Logging").
However, the logs are retained in the dbfs indefinitely, even after 60 days (which is the retention period for job runs history). Is it possible to configure a clean up policy on logs older than a defined threshold via the databricks workspace? If it's not possible to achieve this within databricks, what would be a best practice to do this on Azure?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2023 08:26 AM
@EDDatabricks Thanks for reaching out to us.
I think you should explore our Purge option is enabled or not.
https://docs.databricks.com/en/administration-guide/workspace/settings/storage.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2024 10:52 AM
Pretty sure that controls what's available to the UI in terms of cluster logs, not what resides in the cluster-logs storage directory. Those are two different considerations.
