topic Re: can anyone help with Spill Question in Data Engineering

can anyone help with Spill Question

Aviral-Bhardwaj — Sun, 21 May 2023 10:12:08 GMT

Spill occurs as a result of executing various wide transformations. However, diagnosing a spill requires one to proactively look for key indicators.

Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?

a- Executor's detail screen and Executor's log files

b- Driver's and Executor's log files

c- Stage's detail screen and Query's detail screen

d- Query's detail screen and Job's detail screen

e- Stage's detail screen and Executor's log files

@Hubert Dudek

pvignesh92 — Mon, 22 May 2023 07:30:10 GMT

@Aviral Bhardwaj I feel it is Option e. Stage and executor log files.

Consolidated details at the Stage Level

Details at the task and Executor Level

Please let me know if you feel any other option is better.

Aviral-Bhardwaj — Mon, 22 May 2023 09:29:18 GMT

Exactly Right