- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2022 09:57 AM
Hi @Uma Maheswara Rao Desula
I have written this code, as I have many files in many folders in same location and everything is UTF-16. This is giving me proper result as below
dff = spark.read.option("header", "true") \
.option("inferSchema", "true") \
.option('encoding', 'UTF-16') \
.option('multiline', 'true') \
.option("delimiter", "‡‡,‡‡") \
.csv("/mnt/data/file.csv")
display(dff)
‡‡CompanyId Companyname CountryId‡‡
‡‡1234 abc cn‡‡
‡‡2345 def us‡‡
‡‡3457 ghi sy‡‡
‡‡7564 lmn uk‡‡
Now, I want to remove the start and end double daggers and I wrote below code and it is giving me error "IndentationError: expected an indented block"
from pyspark.sql.functions import regexp_replace
dffs_headers = dff.dtypes
for i in dffs_headers:
columnLabel = i[0]
newColumnLabel = columnLabel.replace('‡‡','').replace('‡‡','')
dff = dff.withColumn(newColumnLabel, regexp_replace(columnLabel, '^\\‡‡|\\‡‡$', ''))
if columnLabel != newColumnLabel:
dff = dff.drop(columnLabel)
display(dff)
error "IndentationError: expected an indented block"