https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/19096#M12753 <P>The error messages mean the Spark driver is having insufficient resources. Although increasing the driver memory or using a bigger instance type is a quick workaround, identifying the issue is the key thing. </P><P>On the application side , review if there are operations which is driver intensive. Say, collect() or toPandas() etc. </P><P>Also check if the instance type used is adequate for the workload. </P><P></P> Fri, 25 Jun 2021 18:47:58 GMT brickster_2018 2021-06-25T18:47:58Z https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/19095#M12752 <P>My job fails with Driver is temporarily unavailable. Apparently, it's permanently unavailable, because the job is not pausing but failing. </P> Fri, 25 Jun 2021 18:43:48 GMT https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/19095#M12752 brickster_2018 2021-06-25T18:43:48Z https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/19096#M12753 <P>The error messages mean the Spark driver is having insufficient resources. Although increasing the driver memory or using a bigger instance type is a quick workaround, identifying the issue is the key thing. </P><P>On the application side , review if there are operations which is driver intensive. Say, collect() or toPandas() etc. </P><P>Also check if the instance type used is adequate for the workload. </P><P></P> Fri, 25 Jun 2021 18:47:58 GMT https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/19096#M12753 brickster_2018 2021-06-25T18:47:58Z https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/39891#M27087 <P>I am facing the same issues .&nbsp; I am writing in batches using a simple for loop. I don't have any collect statements inside the loop. I am rewriting the partitions with partition overwrite dynamic mode in a huge wide delta table - several tb. The incremental load is sometimes 1-2 tb</P> Mon, 14 Aug 2023 20:10:17 GMT https://community.databricks.com/t5/data-engineering/the-driver-is-temporarily-unavailable/m-p/39891#M27087 Chalki 2023-08-14T20:10:17Z