@Atanu Sarkar​ ,

I m using the below shell script for trying to schedule the crontab in databricks node.

crontab-setup-for-log-cleanup.sh

sudo -H -u root bash -c 'echo "$(echo '* */3 * * * sh /dbfs/FileStore/driver-logs-cleanup.sh' ; crontab -l 2>&1)" | crontab -'

driver-logs-cleanup.sh

#!/bin/bash

#Find the the older stdout, stderr and log4j files and deletes it accordingly.

find /dbfs/cluster-logs/<db-cluster-id>/driver/ -name "stdout--*" -exec rm -f {} \;

find /dbfs/cluster-logs/<db-cluster-id>/driver/ -name "stderr--*" -exec rm -f {} \;

find /dbfs/cluster-logs/<db-cluster-id>/driver/ -name "log4j""-""*"".gz" -exec rm -f {} \;

Thanks & Regards,

Sugumar Srinivasan.