Hi @aleksandra_ch,
Unfortunately, the Volumes directory is only accessible through the Databricks interface. At the OS level, it is not accessible, even when running as root (FUSE limitation).
I have also observed intermittent periods where, for certain job runs, logs are not updated in Volumes at all, specifically they are not copied from /databricks/driver/logs to Volumes.
To mitigate this, I implemented an alternative approach that performs direct log synchronization to S3 (the same backing location used by the Volume) at a predefined interval.
It would be very helpful to have either:
- A shutdown_script feature as an alternative to init_script, or
- A built-in log synchronization process executed automatically before cluster termination.
Please find the script used below:
#!/bin/bash
set -e
# ===============================================================
# 1. BASIC CONFIGURATION (S3)
# ===============================================================
S3_BUCKET="<<bucket>>"
S3_BASE_PREFIX="data/<<catalog>>/<<schemas>>/__unitystorage/schemas/<<schema_id>>/volumes/<<volume_id>>"
# Automatically fetch cluster metadata at startup
CLUSTER_ID="${DB_CLUSTER_ID:-unknown_cluster}"
DT=$(date +%Y-%m-%d)
HH=$(date +%H)
MM=$(date +%M)
# ===============================================================
# 2. CREATE THE PYTHON UTILITY FOR DIRECT S3 SYNC
# ===============================================================
cat << 'EOF' > /usr/local/bin/sync_single_run.py
import boto3
import os
import sys
bucket_name = sys.argv[1]
base_prefix = sys.argv[2]
cluster_id = sys.argv[3]
dt = sys.argv[4]
hh = sys.argv[5]
mm = sys.argv[6]
s3_target_prefix = f"{base_prefix}/{cluster_id}/driver"
local_log_dir = "/databricks/driver/logs"
log_files = ["stdout", "stderr", "log4j-active.log", "stacktrace.log"]
s3 = boto3.client('s3')
def get_s3_name(file_name):
if file_name == "log4j-active.log": return f"log4j-{dt}-{hh}-{mm}.log"
if file_name == "stacktrace.log": return f"{dt}-{hh}-{mm}.stacktrace.log"
if file_name == "stdout": return f"stdout--{dt}--{hh}-{mm}.log"
if file_name == "stderr": return f"stderr--{dt}--{hh}-{mm}.log"
return file_name
for file_name in log_files:
local_path = os.path.join(local_log_dir, file_name)
if os.path.exists(local_path) and os.path.getsize(local_path) > 0:
s3_key = f"{s3_target_prefix}/{get_s3_name(file_name)}"
try:
s3.upload_file(local_path, bucket_name, s3_key)
except:
pass # Ignore temporary errors to avoid blocking script execution
EOF
chmod +x /usr/local/bin/sync_single_run.py
# ===============================================================
# 3. CREATE AND START THE BASH DAEMON (LIVE SYNC WATCHER)
# ===============================================================
cat << EOF > /tmp/run_daemon.sh
#!/bin/bash
# Let the cluster finish its initial boot phase before the first sync
sleep 30
while true; do
python3 /usr/local/bin/sync_single_run.py "$S3_BUCKET" "$S3_BASE_PREFIX" "$CLUSTER_ID" "$DT" "$HH" "$MM" > /dev/null 2>&1
sleep 300 # Run every 300 seconds
done
EOF
chmod +x /tmp/run_daemon.sh
# Launch the daemon in the background as an independent process
nohup /bin/bash /tmp/run_daemon.sh > /dev/null 2>&1 &
echo "The permanent Live Sync system (10s) with Databricks pattern was installed successfully!"
