Def running into small file problems. 70gb in 20mbg chunks is 3500 files to read and process so tons of scheduling and overhead.

Adding the year and month columns won't actually increase the data size because those values are captured in the path and not in the actual file.