cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Cluster termination using python script, sending return code 1

Harsh-dataB
New Contributor II

i have used a cluster termination logic for terminating a cluster , the issue is , the cluster is not terminating gracefully , returns a return/exit code 1

The cluster is completing all the spark jobs, but it goes on long running state, hence i created a logic for terminating the clusters getting in long run state,

is there any way to change the return code to 0 , or terminate the cluster gracefully .

2 REPLIES 2

Kaniz
Community Manager
Community Manager

Hi @Harsh-dataB,

  • First, review your cluster termination logic. Make sure it accounts for all necessary cleanup tasks and allows sufficient time for Spark jobs to complete.
  • If you’re using custom scripts or logic, ensure that it gracefully handles all necessary shutdown procedures.
  • Check your cluster configuration settings in Databricks. Verify that resource limits, timeouts, and eviction policies are appropriately configured.
  • Adjust these settings if needed to allow for graceful termination.
  • Unfortunately, Databricks doesn’t directly allow changing the exit code from 1 to 0 during cluster termination.
  • However, you can create a custom script or wrapper around your termination logic. In this script, you can explicitly exit with code 0 after ensuring all necessary cleanup tasks are completed.
  • As a workaround, consider manually terminating the cluster using the Databricks UI or API. This way, you can ensure proper cleanup.
  • If you’re using the Databricks REST API, you can issue a POST request to the /api/2.0/clusters/delete endpoint with the appropriate cluster-ID.

If you encounter any challenges or need further assistance, feel free to ask! 😊🚀

Harsh-dataB
New Contributor II

I am terminating the cluster manually using Databricks API, that only gives the return code 1, and have checked all the necessary changes, but nothing works.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!