Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-17-2023 07:01 AM
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.
ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most recent failure: Lost task 7.4 in stage 481.0 (TID 4116) (10.154.64.26 executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 177186 ms
and also wanted to know the best order of using
VACUUM table RETAIN 168 HOURS
optimize table;
fsck repair table table;
REFRESH table table;
- Labels:
-
Apache spark
-
Executor Logs
-
VACUUM Command
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2023 11:39 AM
Do you have access to the Executor 7 logs? is there a high GC or some other events that is making the heartbeat timeout? would you be able to check the failed stages?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-05-2025 09:28 AM
In my opinion Best order: Optimize → Vacuum → FSCK Repair → Refresh.
Your error is likely a timeout — try more cluster resources or a longer retention period.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thursday
The executor heartbeat timeout makes sense to investigate first, especially with a table that has grown to around 1.5 crore rows. I’d be interested to know whether increasing executor resources actually resolved the issue before changing the command sequence. Also, does running OPTIMIZE before VACUUM make a noticeable difference for a table of this size?
