Loaded a csv file with five columns into a dataframe, and then added around 15+ columns using dataframe.withColumn method.

After adding these many columns, when I run the query df.rdd.isEmpty() - which throws the below error.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 32.0 failed 4 times, most recent failure: Lost task 0.3 in stage 32.0 (TID 28) (10.139.64.4 executor 9): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

Any idea what is the issue?