Thanks for the reply @Miklos Christine​ 

I am unable to see the error in the failed task from the sparkUI and I am preety sure that the schema is perfectly matching the data as I am able to some other stuff correctly querying the data.

Could you please let me know on how to add error handling while converting to a dataframe before writing out to parquet or guide me to the correct resource to find it.

Thanks.