Hello @mkwparth
Like my friend @szymon_dybczak shared the details already.
To dig further, (unfortunately I only have databricks community edition which has resistriction to show metrics) But
Like in the metrics, on the right side - you can select driver dropdown instead of Compute. Just to confirm what is the usage.
You can also run:
%sh
free -h
top -b -n 1 | head -n 20
jps -l
the above are system monitoring commands. Also you can run
%python
import os
pid = os.popen("jps | grep DriverDaemon | awk '{print $1}'").read().strip()
heap_dump_command = f"jmap -dump:live,format=b,file=/tmp/heapdump_{pid}.hprof {pid}"
print(os.popen(heap_dump_command).read())
gc_stats_command = f"jstat -gc {pid}"
print(os.popen(gc_stats_command).read())
memory_summary_command = f"jmap -heap {pid}"
print(os.popen(memory_summary_command).read())
open_files_command = "lsof | grep java | wc -l"
print(os.popen(open_files_command).read())
the above are snapshot of the JVM (Java Virtual Machine) memory utilization for your Spark Driver. You will get much more clear idea about what is actually happening on your driver.
As from the images shared from you, its clear that there is no space available. Like you know, spark in general purpose in memory compute engine, it will try to bring the data from disk(data lake) to memory and do the process.
In this duration, as the compute you selected is using almost the memory, you have to reconfigure the compute for the data you are processing. If you dont do that in the future you will obviously get DRIVER_NOT_RESPONDING Compute Events (Driver is up but is not responsive, likely due to GC).
So
The driver instance type is not optimal for the load executed on the driver.
There are memory-intensive operations executed on the driver.
Many notebooks or jobs are running in parallel on the same cluster. We don't recommend this, because it can cause unexpected behaviors.
I hope this helps you.
.png){kind=link}
.png){kind=link}
