Thanks Lleido. I eventually found I had changed the schema of a partitioned DataFrame that I had made inadvertently where I narrowed a column's type from a long to an integer. While rather obvious cause of the problem in hindsight it was terribly difficult to troubleshoot at first since it was inconsistent. In the end, I found if I turn on "mergeSchema" option to force the schema to merge across all partitions it would at least manifest the problem immediately rather than intermittently as different parts of the code ran (and accessed different old saved data frames). To merge the schema:

sqlContext.read.option('mergeSchema', True).parquet(...)