Hello, I assume you would be using spark to read data from bronze/silver layer to transform and calculate the summarized/aggregated tables and load it into gold layer. Additionally, using delta tables especially in gold layer for incremental upserts ...
Some important info to look in Gangalia UI in CPU, memory and server load charts to spot the problem:CPU chart :User %Idle %High percentage of user % indicates heavy CPU usage in the cluster.Memory chart : Use %Free %Swap % If you see purple line ove...
