Hi @aleksandra_ch,

Unfortunately, the Volumes directory is only accessible through the Databricks interface. At the OS level, it is not accessible, even when running as root (FUSE limitation).

I have also observed intermittent periods where, for certain job runs, logs are not updated in Volumes at all, specifically they are not copied from /databricks/driver/logs to Volumes.
To mitigate this, I implemented an alternative approach that performs direct log synchronization to S3 (the same backing location used by the Volume) at a predefined interval.

It would be very helpful to have either:

1. A shutdown_script feature as an alternative to init_script, or
2. A built-in log synchronization process executed automatically before cluster termination.
   
   Please find the script used below:

#!/bin/bash
set -e

# ===============================================================
# 1. BASIC CONFIGURATION (S3)
# ===============================================================
S3_BUCKET="<<bucket>>"
S3_BASE_PREFIX="data/<<catalog>>/<<schemas>>/__unitystorage/schemas/<<schema_id>>/volumes/<<volume_id>>"

# Automatically fetch cluster metadata at startup
CLUSTER_ID="${DB_CLUSTER_ID:-unknown_cluster}"
DT=$(date +%Y-%m-%d)
HH=$(date +%H)
MM=$(date +%M)

# ===============================================================
# 2. CREATE THE PYTHON UTILITY FOR DIRECT S3 SYNC
# ===============================================================
cat << 'EOF' > /usr/local/bin/sync_single_run.py
import boto3
import os
import sys

bucket_name = sys.argv[1]
base_prefix = sys.argv[2]
cluster_id = sys.argv[3]
dt = sys.argv[4]
hh = sys.argv[5]
mm = sys.argv[6]

s3_target_prefix = f"{base_prefix}/{cluster_id}/driver"
local_log_dir = "/databricks/driver/logs"
log_files = ["stdout", "stderr", "log4j-active.log", "stacktrace.log"]

s3 = boto3.client('s3')

def get_s3_name(file_name):
    if file_name == "log4j-active.log": return f"log4j-{dt}-{hh}-{mm}.log"
    if file_name == "stacktrace.log": return f"{dt}-{hh}-{mm}.stacktrace.log"
    if file_name == "stdout": return f"stdout--{dt}--{hh}-{mm}.log"
    if file_name == "stderr": return f"stderr--{dt}--{hh}-{mm}.log"
    return file_name

for file_name in log_files:
    local_path = os.path.join(local_log_dir, file_name)
    if os.path.exists(local_path) and os.path.getsize(local_path) > 0:
        s3_key = f"{s3_target_prefix}/{get_s3_name(file_name)}"
        try:
            s3.upload_file(local_path, bucket_name, s3_key)
        except:
            pass  # Ignore temporary errors to avoid blocking script execution
EOF

chmod +x /usr/local/bin/sync_single_run.py

# ===============================================================
# 3. CREATE AND START THE BASH DAEMON (LIVE SYNC WATCHER)
# ===============================================================
cat << EOF > /tmp/run_daemon.sh
#!/bin/bash
# Let the cluster finish its initial boot phase before the first sync
sleep 30

while true; do
  python3 /usr/local/bin/sync_single_run.py "$S3_BUCKET" "$S3_BASE_PREFIX" "$CLUSTER_ID" "$DT" "$HH" "$MM" > /dev/null 2>&1
    sleep 300  # Run every 300 seconds
done
EOF

chmod +x /tmp/run_daemon.sh

# Launch the daemon in the background as an independent process
nohup /bin/bash /tmp/run_daemon.sh > /dev/null 2>&1 &

echo "The permanent Live Sync system (10s) with Databricks pattern was installed successfully!"​