Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2022 10:48 PM
Dear @Uma Maheswara Rao Desula Thanks for your help. I am able to split into columns.I have almost 90 columns. So, is there any way that I can automate?
I tried below code, but it is not giving clean seperated columns as ur code.
dff = spark.read.option("header", "true").option("inferSchema", "true").option("delimiter", "‡‡,‡‡").csv(filepath)
dffs_headers = dff.dtypes
for i in dffs_headers:
columnLabel = i[0]
newColumnLabel = columnLabel.replace('‡‡','').replace('‡‡','')
dff = dff.withColumn(newColumnLabel, regexp_replace(columnLabel, '^\\‡‡|\\‡‡$', ''))
if columnLabel != newColumnLabel:
dff = dff.drop(columnLabel)