cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Removing non-ascii and special character in pyspark

RohiniMathur
New Contributor II

i am running spark 2.4.4 with python 2.7 and IDE is pycharm.

The Input file (.csv) contain encoded value in some column like given below.

File data looks

COL1,COL2,COL3,COL4

CM, 503004, (d$όνυ$F|'.h*Λ!ψμ=(.ξ; ,.ʽ|!3-2-704

The output i am trying to get is

CM,503004,,3-2-704 ---- all encoded and ascii value removed.

code i tried :

from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Python Spark").getOrCreate() df = spark.read.csv("filepath\Customers_v01.csv",header=True,sep=","); myres = df.rdd.map(lambda x: x[1].encode().decode('utf-8')) print(myres.collect())

but this is giving only

503004 -- printing only col2 value.

Please share your suggestion , is it possible to fix the issue in pyspark.

Thanks a lot

4 REPLIES 4

shyam_9
Valued Contributor
Valued Contributor

Hi @Rohini Mathur, use below code on column containing non-ascii and special characters.

df['column_name'].str.encode('ascii', 'ignore').str.decode('ascii')

@Shyamprasad Miryala​ : Thanks a lot... can we define multiple column in column name with comma ','

@Shyamprasad Miryala​ : i did like this myres=df['COLC'].str.encode('ascii', 'ignore').str.decode('ascii') but getting error like pyspark.sql.utils.AnalysisException: u'Cannot resolve column name "" among (colA, (colB, (colC);'. please help

It was because of the incorrect structure of the CSV file. Remove the white spaces from the CSV file. Maybe some of the column names contain white spaces before the name itself.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.