Hi werners,

I need to perform the cleanup of azure data bricks driver logs (std.out, std.err, log4j) from dbfs path every hour. to achieve this I'm trying to schedule one Cron job on data bricks driver node so that logs can be deleted every one hour. While using below script in init, the azure databricks cluster creation is failing.

The shell script contains below content:

dbfs:/FileStore/crontab-setup-for-log-cleanup.sh

sudo -H -u root bash -c 'echo "$(echo "* */5 * * * sh /dbfs/FileStore/driver-logs-cleanup.sh" ; crontab -l 2>&1)" | crontab -'