Hi @Eduard, The errors you're experiencing could be due to a few reasons:
1. **Lost communication with the driver node**: This error usually occurs due to networking errors or malfunctioning instances. It might be that the driver node is losing its connection due to network instability or because the instance it's running on is having issues.
2. **Setting up 6 nodes**: This error can occur if there's an issue with the autoscaling process. When you provide a range for the number of workers, Databricks chooses the appropriate number of workers required to run your job. You might see this error if there's a problem during this process.
Here are a few things you could do:
• Check the network stability: Network issues can often cause problems with the driver node. Make sure that your network connection is stable and reliable.
• Check the health of your instances: If you're using cloud instances, check for any issues with them. For example, they might be running out of memory or CPU.
• Check your autoscaling configuration: Ensure you've correctly configured your cluster's minimum and maximum number of workers. Suppose the cluster size exceeds the minimum number of workers selected when the cloud provider terminates instances. In that case, Databricks will continuously retry to re-provision issues to maintain the minimum number of workers.
• Check your workload: Autoscaling works best for workloads whose requirements change over time. If your workload is consistently high or low, you might be better off with a fixed number of workers.
Sources:
- [Cluster size and autoscaling](https://docs.databricks.com/archive/compute/configure.html)
- [How autoscaling behaves](https://docs.databricks.com/clusters/configure.html)
