Performance Issues experienced when cluster was upgraded from 10.4 LTS to 11.3 LTS

[Shivani_DB]

Performance Issues experienced when cluster was upgraded from 10.4 LTS to 11.3 LTS , The notebooks were running fine with the existing cluster. Soon after the upgrade the notebooks started to fail or exhaust memory executors etc . Any suggestions?

[Kaniz_Fatma]

Hi @Shivani_DB,  The possible reasons for the issues might be:
 - Incompatible dependencies with Spark elastic jar
 - Class not found error: org/sparkproject/guava/cache/CacheLoader

Suggestions to resolve the issues:
 - Check for incompatible dependencies in code and update or modify accordingly
 - Verify cluster configurations, especially Spark Session Isolation
 - Monitor cluster performance using tools like Ganglia
 - Increase cluster capacity if notebooks are failing or memory executors are exhausted
 - Check the driver log for any issues