cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Is it safe to turn off pre-emption on an on-demand cluster?

User16869510359
Esteemed Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

Pre-emption is by default turned on Databricks cluster. Turning on or turning off pre-emption would make more sense on a high concurrency cluster. Pre-emption ensures that the job starting for resources gets a fair share of the resource available on the cluster. To make the resources available, pre-emption would kill the running tasks. 

To answer the question, as there are no concurrent jobs on an on-demand cluster, it would not make much difference if the pre-emption is enabled or disabled. In case if there are concurrent jobs submitted using internal application logic, then there will be an impact. 

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

Pre-emption is by default turned on Databricks cluster. Turning on or turning off pre-emption would make more sense on a high concurrency cluster. Pre-emption ensures that the job starting for resources gets a fair share of the resource available on the cluster. To make the resources available, pre-emption would kill the running tasks. 

To answer the question, as there are no concurrent jobs on an on-demand cluster, it would not make much difference if the pre-emption is enabled or disabled. In case if there are concurrent jobs submitted using internal application logic, then there will be an impact. 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.