Hi @Yeshwanth, thank you for directing me to the documentation. I don't know much about computations, so I'm still figuring things out. So is there like a straight forward (standard) way to calculate the compute (no. of cores & memory) required to run spark jobs based on certain data volume of the job, frequency of the jobs, and number of jobs? I read that the data is generally partitioned into 128MB and the executor memory is divided into 300 MB reserved memory, 60% execution memory, and 40% storage memory. How would this help me calculate the compute for a data of size, say 1.5 TB?
