topic I need to edit my parquet files, and change field name, replacing space by underscore in Data Engineering

I need to edit my parquet files, and change field name, replacing space by underscore

prakharjain — Mon, 02 Mar 2020 18:34:16 GMT

Hello,

I am facing trouble as mentioned in following topics in stackoverflow,

https://stackoverflow.com/questions/45804534/pyspark-org-apache-spark-sql-analysisexception-attribute-name-contains-inv

https://stackoverflow.com/questions/38191157/spark-dataframe-validating-column-names-for-parquet-writes-scala

I have tried all the solution mentioned there, but I am getting same error every time. Its like spark cannot read fields with space in them.

So, I am trying to find any other solution just to rename my fields, and save the parquet files back. After that I will continue my transformation with spark.

Anyone can help me out.. Loads of love and thanks 🙂

Re: I need to edit my parquet files, and change field name, replacing space by underscore

DimitriBlyumin — Wed, 13 May 2020 11:38:34 GMT

Looks like it is a known issue/limitation due to Parquet internals, and it will not be fixed. Apparently there is no workaround in Spark.

https://issues.apache.org/jira/browse/SPARK-27442

Re: I need to edit my parquet files, and change field name, replacing space by underscore

DimitriBlyumin — Thu, 21 May 2020 11:48:22 GMT

One option is to use something other than Spark to read the problematic file, e.g. Pandas, if your file is small enough to fit on the driver node (Pandas will only run on the driver). If you have multiple files - you can loop through them and fix one-by-one.

import pandas as pd
df = pd.read_parquet('//dbfs/path/to/your/file.parquet')
df = df.rename(columns={
  "Column One" : "col_one", 
  "Column Two" : "col_two"
})
dfSpark = spark.createDataFrame(df) # convert to Spark dataframe
df.to_parquet('//dbfs/path/to/your/fixed/file.parquet') # and/or save fixed Parquet