To make it feasible to recover/schema change for instances like this in the future, I save saving all dataframes into a subfolder with the "/key=..." naming convention in the file name. So the filename looks like the following:
path = METRICS_DATA_...
In that particular case, I think I decided the data wasn't feasible to recover. It might be, but I decided to stop spending time on it and rebuilt the data from source. It wasn't a terribly large amount of data so that was feasible in that case.
In...
Thanks Lleido. I eventually found I had changed the schema of a partitioned DataFrame that I had made inadvertently where I narrowed a column's type from a long to an integer. While rather obvious cause of the problem in hindsight it was terribly di...
