Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2024 11:29 AM
Yes thats the issue I'm facing, Im so confused about this because, do i have to rename the child columns? I just want to keep the spark dataframe as it is without flattening, . and if i do this it only updates the parent column names:
def clean_column_names(df😞
for col in df.columns:
new_col_name = col.replace('$', '').replace('.', '_').replace(' ', '_').replace(';', '_').replace(',', '_')
df = df.withColumnRenamed(col, new_col_name)
return df
df_cleaned = clean_column_names(df)
# Check the cleaned column names
df_cleaned.printSchema()
df_cleaned.columns
