Cluster termination using python script, sending return code 1

[Harsh-dataB]

i have used a cluster termination logic for terminating a cluster , the issue is , the cluster is not terminating gracefully , returns a return/exit code 1

The cluster is completing all the spark jobs, but it goes on long running state, hence i created a logic for terminating the clusters getting in long run state,

is there any way to change the return code to 0 , or terminate the cluster gracefully .

[Kaniz_Fatma]

Hi @Harsh-dataB,

• First, review your cluster termination logic. Make sure it accounts for all necessary cleanup tasks and allows sufficient time for Spark jobs to complete.
• If you’re using custom scripts or logic, ensure that it gracefully handles all necessary shutdown procedures.
• Check your cluster configuration settings in Databricks. Verify that resource limits, timeouts, and eviction policies are appropriately configured.
• Adjust these settings if needed to allow for graceful termination.
• Unfortunately, Databricks doesn’t directly allow changing the exit code from 1 to 0 during cluster termination.
• However, you can create a custom script or wrapper around your termination logic. In this script, you can explicitly exit with code 0 after ensuring all necessary cleanup tasks are completed.
• As a workaround, consider manually terminating the cluster using the Databricks UI or API. This way, you can ensure proper cleanup.
• If you’re using the Databricks REST API, you can issue a POST request to the /api/2.0/clusters/delete endpoint with the appropriate cluster-ID.

If you encounter any challenges or need further assistance, feel free to ask! 😊🚀

[Harsh-dataB]

I am terminating the cluster manually using Databricks API, that only gives the return code 1, and have checked all the necessary changes, but nothing works.