I have a set of CSV files generated by a system, where the schema has evolved over the years. Some columns have been added, and at least one column has been renamed in newer files. Is there any way to elegantly load these files into a dataframe?

I have tried spark.read.csv() using different options. My next thought would be to load the individual files using pandas, possibly using applyInPandas.

Any thoughts or ideas?