Hi @Nagarathna, a couple of questions here...

1. What error is the job returning after the 2 hours execution?
2. What is the data source we are talking about? Are you trying to load data from a database, another warehouse, a bunch of csv/json/excel files?
3. How are you chunking the data? (if you could give a code example...)
4. Are there any resource bottlenecks visible in the Spark UI (e.g., memory, CPU, disk, network)?
5. Do you see any relevant warnings or errors in the Spark executor/driver logs before the failure?
6. The cluster is fix sized with 4 workers or auto-scale is enabled?