Tharun-Kumar
Databricks Employee
Databricks Employee

@Magnus 

  • You can read the input file using Pandas or Koalas (https://koalas.readthedocs.io/en/latest/index.html)
  • then rename the columns
  • then convert the Pandas/Koalas dataframe to Spark dataframe. You can write it back with the correct column name, so the next time you use it, the error will not happen.