How to enforce a cleanup policy on job cluster logs
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2023 07:07 AM
We have a number of jobs on our databricks workspaces. All job clusters are configured with a dbfs location to save the respective logs (configured from Job cluster -> "Advanced options" -> "Logging").
However, the logs are retained in the dbfs indefinitely, even after 60 days (which is the retention period for job runs history). Is it possible to configure a clean up policy on logs older than a defined threshold via the databricks workspace? If it's not possible to achieve this within databricks, what would be a best practice to do this on Azure?