cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Cluster xxxxxxx was terminated during the run.

Eduard
New Contributor II

Hello,

I have a problem with the autoscaling of a cluster. Every time the autoscaling is activated I get this error. Does anyone have any idea why this could be?

"Cluster xxxxxxx was terminated during the run (cluster state message: Lost communication with the driver node. This can occur because of networking errors or malfunctioning instances. databricks_error_message: driver is lost) "

Also from time to time I get this error also: 

Cluster xxxxxx  was terminated during the run (cluster state message: Setting up 6 nodes.)

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @EduardThe errors you're experiencing could be due to a few reasons:

1. **Lost communication with the driver node**: This error usually occurs due to networking errors or malfunctioning instances. It might be that the driver node is losing its connection due to network instability or because the instance it's running on is having issues.

2. **Setting up 6 nodes**: This error can occur if there's an issue with the autoscaling process. When you provide a range for the number of workers, Databricks chooses the appropriate number of workers required to run your job. You might see this error if there's a problem during this process.

Here are a few things you could do:

• Check the network stability: Network issues can often cause problems with the driver node. Make sure that your network connection is stable and reliable.

• Check the health of your instances: If you're using cloud instances, check for any issues with them. For example, they might be running out of memory or CPU.

• Check your autoscaling configuration: Ensure you've correctly configured your cluster's minimum and maximum number of workers. Suppose the cluster size exceeds the minimum number of workers selected when the cloud provider terminates instances. In that case, Databricks will continuously retry to re-provision issues to maintain the minimum number of workers.

• Check your workload: Autoscaling works best for workloads whose requirements change over time. If your workload is consistently high or low, you might be better off with a fixed number of workers.

Sources:
- [Cluster size and autoscaling](https://docs.databricks.com/archive/compute/configure.html)
- [How autoscaling behaves](https://docs.databricks.com/clusters/configure.html)

Eduard
New Contributor II

So i could see more in deep the logs and i got this:

CPU is not the problem. 

Caused by: com.databricks.backend.manager.instance.FirewallSetupException: Fail to setup inbound Firewall.

 

I got this error while the autoscaling was ON. Must be something with my network, not sure what.. 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!