cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Data bricks cluster creation is failing while running the Cron job scheduling script through init script method from azure data bricks.

Sugumar_Sriniva
New Contributor III

Dear connections,

I'm unable to run a shell script which contains scheduling a Cron job through init script method on Azure Data bricks cluster nodes.

Error from Azure Data bricks workspace:

"databricks_error_message": "Cluster scoped init script dbfs:/<shell-script> failed: Script exit status is non-zero"

Looking for a method to achieve the same.

1 ACCEPTED SOLUTION

Accepted Solutions

User16764241763
Honored Contributor

Hello @Sugumar Srinivasanโ€‹  Could you please enable cluster log delivery and inspect the INIT script logs in the below path dbfs:/cluster-logs/<clusterId>/init_scripts path.

https://docs.databricks.com/clusters/configure.html#cluster-log-delivery-1

View solution in original post

11 REPLIES 11

Anonymous
Not applicable

Hello @Sugumar Srinivasanโ€‹ - Welcome to the community! It's nice to meet you. My name is Piper, and I'm a moderator for Databricks.

Let's give the other members a chance to answer your question. We'll come back to this if we need to. ๐Ÿ™‚

-werners-
Esteemed Contributor III

I do not completely understand what you are trying to do?

Are you trying to schedule a cron job on the cluster workers?

Sugumar_Sriniva
New Contributor III

Hi werners,

I need to perform the cleanup of azure data bricks driver logs (std.out, std.err, log4j) from dbfs path every hour. to achieve this I'm trying to schedule one Cron job on data bricks driver node so that logs can be deleted every one hour. While using below script in init, the azure databricks cluster creation is failing.

The shell script contains below content:

dbfs:/FileStore/crontab-setup-for-log-cleanup.sh

sudo -H -u root bash -c 'echo "$(echo "* */5 * * * sh /dbfs/FileStore/driver-logs-cleanup.sh" ; crontab -l 2>&1)" | crontab -'

-werners-
Esteemed Contributor III

Does the script work while the cluster is running, to exclude a issue with the script itself?

Yes I'm able to make a cron entry in the running cluster's spark driver node through ssh. โ€‹

Tasootoor1998
New Contributor III

DataBricks Cluster Creation is failing while running the cron job scheduling script through init Script Method from Azure Data Bricks

Atanu
Databricks Employee
Databricks Employee

the issue is definitely the init script.Please cross check the init script or you can post it here if no sensitive info. we can cross verify.

Atanu
Databricks Employee
Databricks Employee

@Sugumar Srinivasanโ€‹  are you able to check the init script to rectify the issue?

Sugumar_Sriniva
New Contributor III

@Atanu Sarkarโ€‹ ,

I m using the below shell script for trying to schedule the crontab in databricks node.

crontab-setup-for-log-cleanup.sh

sudo -H -u root bash -c 'echo "$(echo '* */3 * * * sh /dbfs/FileStore/driver-logs-cleanup.sh' ; crontab -l 2>&1)" | crontab -'

driver-logs-cleanup.sh

#!/bin/bash

#Find the the older stdout, stderr and log4j files and deletes it accordingly.

find /dbfs/cluster-logs/<db-cluster-id>/driver/ -name "stdout--*" -exec rm -f {} \;

find /dbfs/cluster-logs/<db-cluster-id>/driver/ -name "stderr--*" -exec rm -f {} \;

find /dbfs/cluster-logs/<db-cluster-id>/driver/ -name "log4j""-""*"".gz" -exec rm -f {} \;

Thanks & Regards,

Sugumar Srinivasan.

Hi @Sugumar Srinivasanโ€‹ ,

Just a friendly follow-up. Are you still having issues with your init script or not any more? Please let us know.

User16764241763
Honored Contributor

Hello @Sugumar Srinivasanโ€‹  Could you please enable cluster log delivery and inspect the INIT script logs in the below path dbfs:/cluster-logs/<clusterId>/init_scripts path.

https://docs.databricks.com/clusters/configure.html#cluster-log-delivery-1

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group