Spark feature where to avoid network io it writes each shuffle partition as a 'part...' file on disk and each file as you said will have compression and efficient encoding by default.So Yes it is directly related to parallel processing !!
Can you please check on particular SQL warehouse instances type quota with cloud provider (Azure/AWS) . may be that particular instance type quota is over with your account . Same issue we have faced and requested AWS to increase the quota of that in...
