I have Data Engineering Pipeline workload that run on Databricks.
Job cluster has following configuration :-
Worker i3.4xlarge with 122 GB memory and 16 cores
Driver i3.4xlarge with 122 GB memory and 16 cores ,
Min Worker -4 and Max Worker 8
We noticed that CPU utlization goes higher than 100% few times as a spike.
Can someone help me to understand following questions
1- Are these High CPU Utilization Spikes Problematic
2- Is there any way to check the DBX Job cluster log to see the CPU utilization
3- What is the max limit for CPU utilization and how does this whole things work.
