Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-21-2022 06:48 AM
hello @Salah K. you can try configuring spark.executor.memory from cluster spark configuration.
total_executor_memory = (total_ram_per_node -1) / executor_per_node
total_executor_memory = (64–1)/3 = 21(rounded down)
spark.executor.memory = total_executor_memory * 0.9
spark.executor.memory = 21*0.9 = 18 (rounded down)
memory_overhead = 21*0.1 = 3 (rounded up)