Hi @Saritha_S 
I've updated the Spark configuration as per your suggestion, but I'm still encountering issues.

Out of 6 retry attempts, 4 resulted in the following error:
com.databricks.pipelines.common.errors.deployment.DeploymentException: Communication lost with driver.
The remaining 2 attempts failed due to a REPL timeout error. See the attached screenshots.

Could you please advise on what steps I can take to prevent these errors in the future? Let me know if you need any additional information from my side.

Thanks in advance for your Help!