aleksandra_ch
Databricks Employee
Databricks Employee

Hi @ccsalt ,

Can you try this one:

#!/bin/bash
# Cluster log preservation init script.
#
# Cluster log delivery overwrites stdout/stderr/log4j-active.log/stacktrace.log
# on every cluster restart. This script runs at startup and makes a timestamped
# copy of those files so the previous session's logs are kept.
#
# To use: edit LOG_BASE below to match your cluster's cluster_log_conf destination,
# then attach this script as an init script.

set -uo pipefail

# >>> EDIT THIS to your cluster_log_conf destination <<<
LOG_BASE="/Volumes/<catalog>/<schema>/<volume>/<subdir>"

TS="$(date -u +%Y%m%dT%H%M%SZ)"
DRIVER_LOG_DIR="${LOG_BASE}/${DB_CLUSTER_ID}/driver"

echo "[preserve_logs] cluster=${DB_CLUSTER_ID} ts=${TS}"
echo "[preserve_logs] checking ${DRIVER_LOG_DIR}"

if [ ! -d "${DRIVER_LOG_DIR}" ]; then
  echo "[preserve_logs] no prior driver log dir, nothing to preserve"
  exit 0
fi

for f in stdout stderr log4j-active.log stacktrace.log; do
  src="${DRIVER_LOG_DIR}/${f}"
  dst="${DRIVER_LOG_DIR}/${f}.preserved-${TS}"
  if [ -f "${src}" ]; then
    cp "${src}" "${dst}" && echo "[preserve_logs] preserved ${f} -> $(basename ${dst})"
  fi
done

echo "[preserve_logs] done"

Best regards,