I have a spark pipeline which reads selected data from a table_1 as view and performs few aggregation via group by in next step and writes to target table. table_1 has large data ~30GB, compressed csv.

Step-1:

create or replace temporary view base_data as 
select /* distinct */ a.*, cast(timestamp_field as date) as date...
from table_1 a
where <basic_condition>

Step-2:

select <few columns> ,sum(..),max(..)
from base_data
group by <few columns>

Step-3: Write to target

This works fine. But the input table table_1 contains duplicates. When I enable the DISTINCT clause (currently commented out) in the view, the job fails due to an out-of-memory (OOM) error. Since GROUP BY also involves shuffling, why does adding DISTINCT cause the job to fail? I went through this StackOverflow discussion, but the question isn't about comparing the two when using the same set of columns.