Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-11-2024 08:30 PM - edited 11-11-2024 08:47 PM
Hi,
I have faced this issue a few times. When we are overwriting the dataframes to hive catalog in databricks, it doesn't naturally allow for column names to have spaces or special characters. However, you can add an option statement to bypass that rule and save your dataframes as tables with column names having spaces or special characters.
Solution:
Let's say your dataframe name is 'df'.
The normal pyspark command for saving tables is:
>> df.write.mode("overwrite").format("delta").saveAsTable("catalog.schema_name.table_name")
This will give you the AnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema error.
You need to add an 'option statement' - ".option("delta.columnMapping.mode", "name")" to the above command after overwrite, to resolve it.
The final working command would be:
>> df.write.mode("overwrite").option("delta.columnMapping.mode", "name").format("delta").saveAsTable("catalog.schema_name.table_name")
The above edit, resolves the issue for me and the table is perfectly saved even if it has spaces or special characters in the schema.
Hope it works for you as well. If it does, happy to help!!
KandyKad
