Hi @raghvendrarm1 - Have a look at the section "Apache Spark’s Distributed Execution" in chapter 1 of Learning Spark, 2nd Edition (https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/ch01.html). Have a look at teh picture -

Hello @raghvendrarm1  ,

Below are the answers to your questions:

Do executors always send “results” to the driver?

• No. Only actions that return values (e.g., collect, take, first, count) bring data back to the driver. collect explicitly “returns all the records … as a list” in the driver’s memory and is bounded by spark.driver.maxResultSize. In contrast, writes (df.write…, INSERT, SAVE) are performed by executors to storage, with the driver just coordinating commits. Shuffles move data between executors, not through the driver. 

How is it done under the hood (at a high level)?

• Return-to-driver actions: Each task serialises its partition’s result; the driver gathers them (subject to spark.driver.maxResultSize). That’s why a large collect() can OOM the driver. 
•  Spark’s Data Source API V2 has executors write partition outputs and send small commit messages back; the driver finalizes the job commit.