Saving logs from an all-purpose cluster to Volume or S3 is not consistent, because stderr, stdout, and log4j-active.log get overwritten when the cluster is restarted between minutes 01 and 59.

Tested case:
A job is configured to start every 20 minutes, for example: 10:10 -> 10:30 -> 10:50.
Cluster logs (stderr, stdout, log4j-active.log) are overwritten at each restart, because the cluster does not reach the exact hour (10:00, 11:00) when automatic log rotation happens.

In the Databricks UI, the logs appear later, but with the same name (for example log4j-active.log).
The issue is that, although they seem visible in the UI, in Volume the files are overwritten and information is lost.

Does anyone have an idea of how I can still preserve all logs?

Thanks!