Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2024 01:43 PM
@ivanychev - Firstly, New workers are added and spark notice them hence, there is an init script logging in the event log stating the init script ran on the newly added workers. For debugging, please check the Spark UI - executor tab.
Secondly, For Spot Instance termination, This is mostly by the cloud provider and spot instance price fluctuation. you can ideally use hybrid clusters (with spot fall back on demand) flag set on the cluster configuration page.
Thanks, Shan