- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 11:43 AM
My job fails with Driver is temporarily unavailable. Apparently, it's permanently unavailable, because the job is not pausing but failing.
- Labels:
-
Apache spark
-
Driver
-
Job
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 11:47 AM
The error messages mean the Spark driver is having insufficient resources. Although increasing the driver memory or using a bigger instance type is a quick workaround, identifying the issue is the key thing.
On the application side , review if there are operations which is driver intensive. Say, collect() or toPandas() etc.
Also check if the instance type used is adequate for the workload.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 11:47 AM
The error messages mean the Spark driver is having insufficient resources. Although increasing the driver memory or using a bigger instance type is a quick workaround, identifying the issue is the key thing.
On the application side , review if there are operations which is driver intensive. Say, collect() or toPandas() etc.
Also check if the instance type used is adequate for the workload.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-14-2023 01:10 PM
I am facing the same issues . I am writing in batches using a simple for loop. I don't have any collect statements inside the loop. I am rewriting the partitions with partition overwrite dynamic mode in a huge wide delta table - several tb. The incremental load is sometimes 1-2 tb